All Apps and Add-ons

How to configure the Splunk App for AWS and Splunk Add-on for Amazon Web Services to ingest AWS Config logs?

rarneson
Explorer

Hi,

I'm running the latest Splunk App for AWS (3.0) and Splunk Add-on for Amazon Webservices (1.01).

I'm trying to set it up to ingest my AWS Config logs. There doesn't seem to be a specific "AWS Config" data input, so I'm using the CloudTrail input and choosing my AWS Config SQS Queue. I created an aws-config index and am tagging the events with aws:config.

Here's the errors I'm seeing in /opt/splunk/var/log/splunk/aws_cloudtrail.log

2015-02-11 16:48:50,566 ERROR pid=6635 tid=MainThread file=aws_cloudtrail.py:process_notifications:351 | problems unzipping from s3:config-splunk//: IOError Not a gzipped file
2015-02-11 16:48:50,566 CRITICAL pid=6635 tid=MainThread file=aws_cloudtrail.py:stream_events:286 | Outer catchall: KeyError: 's3Bucket'

I'm a bit stuck as to what the problem is. I've tried two different AWS accounts, one using the root keys. I see my Config > SNS > SQS > S3 data looking correct in AWS. I can even download some of the files from the S3 bucket onto the Splunk server and run file and/or gunzip on it, and everything checks out fine for being a valid gzip with normal json data.

Anyone have any ideas?

Forgot to mention, the ultimate error is nothing is being ingested via the app into my index/populating the dashboards/etc...

0 Karma
1 Solution

_d_
Splunk Employee
Splunk Employee

Couple of things:
- You're using the wrong Add-on version. You'll need 1.1.0 instead of 1.0.1. https://apps.splunk.com/app/1274/#/documentation
- You will need to use the AWS Config input instead of the AWS CloudTrail input to collect Config data.

Update the Add-on to 1.1.0 and you should be fine.

View solution in original post

_d_
Splunk Employee
Splunk Employee

Couple of things:
- You're using the wrong Add-on version. You'll need 1.1.0 instead of 1.0.1. https://apps.splunk.com/app/1274/#/documentation
- You will need to use the AWS Config input instead of the AWS CloudTrail input to collect Config data.

Update the Add-on to 1.1.0 and you should be fine.

rarneson
Explorer

Thank you! I have no idea how I missed that, even checking for it and thinking I was on the latest version, but going back to Manage Apps I saw an upgrade option there.

That was 100% the problem.

Much appreciated, all is working

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

Don't feel bad, you were looking while we were updating 🙂

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...