How to configure the Splunk App for AWS and Splunk Add-on for Amazon Web Services to ingest AWS Config logs?

[rarneson]

Hi,

I'm running the latest Splunk App for AWS (3.0) and Splunk Add-on for Amazon Webservices (1.01).

I'm trying to set it up to ingest my AWS Config logs. There doesn't seem to be a specific "AWS Config" data input, so I'm using the CloudTrail input and choosing my AWS Config SQS Queue. I created an aws-config index and am tagging the events with aws:config.

Here's the errors I'm seeing in /opt/splunk/var/log/splunk/aws_cloudtrail.log

2015-02-11 16:48:50,566 ERROR pid=6635 tid=MainThread file=aws_cloudtrail.py:process_notifications:351 | problems unzipping from s3:config-splunk//: IOError Not a gzipped file
2015-02-11 16:48:50,566 CRITICAL pid=6635 tid=MainThread file=aws_cloudtrail.py:stream_events:286 | Outer catchall: KeyError: 's3Bucket'

I'm a bit stuck as to what the problem is. I've tried two different AWS accounts, one using the root keys. I see my Config > SNS > SQS > S3 data looking correct in AWS. I can even download some of the files from the S3 bucket onto the Splunk server and run file and/or gunzip on it, and everything checks out fine for being a valid gzip with normal json data.

Anyone have any ideas?

Forgot to mention, the ultimate error is nothing is being ingested via the app into my index/populating the dashboards/etc...

[_d_ · Accepted solution]

Couple of things:
- You're using the wrong Add-on version. You'll need 1.1.0 instead of 1.0.1. https://apps.splunk.com/app/1274/#/documentation
- You will need to use the AWS Config input instead of the AWS CloudTrail input to collect Config data.

Update the Add-on to 1.1.0 and you should be fine.

[rarneson]

Thank you! I have no idea how I missed that, even checking for it and thinking I was on the latest version, but going back to Manage Apps I saw an upgrade option there.

That was 100% the problem.

Much appreciated, all is working

[jcoates_splunk]

Don't feel bad, you were looking while we were updating 🙂