All Apps and Add-ons

While trying to set up Splunk using an AMI provided by the AWS marketplace, why is it failing to create a virtual index connection?

amitsri009
New Member

hi,

I'm new to this setup. I used an AMI provided by AWS marketplace for splunk setup.
i created an EMR in AWS and provided details in emr-provider(internal IP of master node:9001 ) for job tracker (since created both server in same subnet)

While creating a virtual index, I provided data in HDFS as (bucket name where elb logs reside with access_id and secret key), but I'm getting this error:

[emr-provider] Error while running external process, return_code=255. See search.log for more info

[emr-provider] RuntimeException - Failed to create a virtual index filesystem connection: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or fs.s3.awsSecretAccessKey properties (respectively).. Advice: Verify that your vix.fs.default.name is correct and available.

Now I am not able to understand where the problem is. Please help me to sort out this problem.

0 Karma

Ledion_Bitincka
Splunk Employee
Splunk Employee

When using AWS our recommendation is to use the OEM/hourly integration which takes care of a lot of the underlying auth issues - more info here and here

Regarding the error message you're seeing it's basically telling you that we cannot connect to S3 to fetch the data. To solve that you need to do one of the following:

a. provide credentials in the vix path (notice using s3n instead of s3)
input path: s3n://[awsAccessKeyId]:[awsSecretAccessKey]@bucket_name/path/to/data

b. provide the credentials in the provider via "Additional settings":

vix.fs.s3n.awsAccessKeyId = 
vix.fs.s3n.awsSecretAccessKey = 

then in your vix input path use s3n://bucket_name/path/to/data
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...