All Apps and Add-ons

Configuring virtual index to reach out to S3 over Direct Connect

mdsnmss
SplunkTrust
SplunkTrust

I'm currently working on setting up a virtual index that connects to an Amazon EMR instance and queries data in S3. It is much like this: https://www.splunk.com/blog/2016/10/30/how-to-splunk-analytics-for-hadoop-on-amazon-web-services-ela.... I have the provider up and working and can search HDFS data directly on the EMR node but run into issues when trying to configure a virtual index to connect to S3. For the "Path to data in HDFS" setting when adding a virtual index I have tried setting the prefix with both S3 and s3n with no luck. I do have the awsAccessKeyId and awsSecretAccessKey for the S3 properly set in hdfs-site.xml.

Looking through search.log when attempting to query the S3 data I can see it attempting to reach out to the S3 path defined in the virtual index but it looks like it fills in data automatically assuming to try reach out to S3 in public AWS. Since we have a Direct Connect (https://aws.amazon.com/directconnect/) is there a way to tell Splunk to go there instead of autofinishing the path with the public domain?

1 Solution

rdagan_splunk
Splunk Employee
Splunk Employee

Below is my working environment using Minio (open source S3)
In addition, here are 2 links that might be helpful:
https://docs.splunk.com/Documentation/Splunk/latest/Indexer/ArchivingSplunkindexestoS3
and
https://answers.splunk.com/answers/457052/how-to-configure-650-data-roll-to-search-archived.html

[provider:minio_provider]
vix.command.arg.3 = $SPLUNK_HOME/bin/jars/SplunkMR-hy2.jar
vix.env.HADOOP_HOME = /opt/hadoop-2.9.2
vix.env.JAVA_HOME = /usr
vix.family = hadoop
vix.fs.default.name = hdfs://127.0.0.1:8020
vix.mapreduce.framework.name = yarn
vix.output.buckets.max.network.bandwidth = 0
vix.splunk.home.hdfs = /user/root/splunkmr
vix.env.HADOOP_TOOLS = $HADOOP_HOME/share/hadoop/common/lib
vix.fs.s3a.access.key = my_access_key
vix.fs.s3a.endpoint = http://127.0.0.1:9000
vix.fs.s3a.impl = org.apache.hadoop.fs.s3a.S3AFileSystem
vix.fs.s3a.path.style.access = true
vix.fs.s3a.secret.key = my_secret_key
vix.mode = stream
vix.splunk.jars = $HADOOP_TOOLS/hadoop-aws-2.9.2.jar,$HADOOP_TOOLS/aws-java-sdk-1.11.234.jar
vix.env.HADOOP_HEAPSIZE = 1024

[minio_test]
vix.input.1.accept = .gz$
vix.input.1.path = s3a://splunkarchive/...
vix.provider = minio_provider

View solution in original post

rdagan_splunk
Splunk Employee
Splunk Employee

Below is my working environment using Minio (open source S3)
In addition, here are 2 links that might be helpful:
https://docs.splunk.com/Documentation/Splunk/latest/Indexer/ArchivingSplunkindexestoS3
and
https://answers.splunk.com/answers/457052/how-to-configure-650-data-roll-to-search-archived.html

[provider:minio_provider]
vix.command.arg.3 = $SPLUNK_HOME/bin/jars/SplunkMR-hy2.jar
vix.env.HADOOP_HOME = /opt/hadoop-2.9.2
vix.env.JAVA_HOME = /usr
vix.family = hadoop
vix.fs.default.name = hdfs://127.0.0.1:8020
vix.mapreduce.framework.name = yarn
vix.output.buckets.max.network.bandwidth = 0
vix.splunk.home.hdfs = /user/root/splunkmr
vix.env.HADOOP_TOOLS = $HADOOP_HOME/share/hadoop/common/lib
vix.fs.s3a.access.key = my_access_key
vix.fs.s3a.endpoint = http://127.0.0.1:9000
vix.fs.s3a.impl = org.apache.hadoop.fs.s3a.S3AFileSystem
vix.fs.s3a.path.style.access = true
vix.fs.s3a.secret.key = my_secret_key
vix.mode = stream
vix.splunk.jars = $HADOOP_TOOLS/hadoop-aws-2.9.2.jar,$HADOOP_TOOLS/aws-java-sdk-1.11.234.jar
vix.env.HADOOP_HEAPSIZE = 1024

[minio_test]
vix.input.1.accept = .gz$
vix.input.1.path = s3a://splunkarchive/...
vix.provider = minio_provider

mdsnmss
SplunkTrust
SplunkTrust

Thanks @rdagan_splunk! The setting I think I was looking for was vix.fs.s3a.endpoint = http://127.0.0.1:9000. It's not something that is listed in the Per Virtual Provider/Index settings of indexes.conf.spec. Doing some research I found it can be set in hdfs-site.xml in your $HADOOP_HOME/etc/hadoop as well.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...