All Apps and Add-ons

Splunk Add-on for Amazon Web Services: Why are we unable to collect AWS cloudtrail data and getting and s3Bucket error?

bendter
Explorer

Splunk Version 6.2.0
Splunk Add-on for AWS version 1.1.0

We have attempted to follow the installation instructions for the Splunk AWS Add-on. We have Cloudtrail enabled dumping to an S3 bucket. We have SNS notifications configured as well as the SQS queue. Messages are showing as available in the SQS queue. We have a Splunk user set up with full access to S3 buckets and Cloudtrail.

On the Splunk side we have the Cloudtrail and Cloudwatch inputs configured. We can successfully receive data from Cloudwatch.

We cannot import any data from Cloudtrail. Does anyone have any ideas what we might be missing?

Here are the log entries:

aws_cloudtrail.log:

2015-01-14 14:48:46,228 INFO pid=14944 tid=MainThread file=aws_cloudtrail.py::409 | STARTED: 
2015-01-14 14:48:46,229 DEBUG pid=14944 tid=MainThread file=aws_cloudtrail.py:stream_events:178 | Start streaming.
2015-01-14 14:48:46,229 DEBUG pid=14944 tid=MainThread file=aws_cloudtrail.py:stream_events:196 | blacklist regex for eventNames is ^(?:Describe|List|Get)
2015-01-14 14:48:46,229 INFO pid=14944 tid=MainThread file=aws_cloudtrail.py:get_access_key_pwd_real:109 | get account name: Splunk General access user
2015-01-14 14:48:46,254 DEBUG pid=14944 tid=MainThread file=aws_cloudtrail.py:stream_events:210 | Connect to S3 & Sqs sucessfully
2015-01-14 14:48:46,563 CRITICAL pid=14944 tid=MainThread file=aws_cloudtrail.py:stream_events:278 | Outer catchall: KeyError: 's3Bucket'
2015-01-14 14:48:46,563 INFO pid=14944 tid=MainThread file=aws_cloudtrail.py::411 | EXITED: 1

splunkd.log:

01-14-2015 14:48:46.008 -0800 INFO  ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/Splunk_TA_aws/bin/aws_cloudwatch.py
01-14-2015 14:48:46.008 -0800 INFO  ExecProcessor -     interval: 30000 ms
01-14-2015 14:48:46.009 -0800 INFO  ExecProcessor - New scheduled exec process: /opt/splunk/bin/splunkd instrument-resource-usage
01-14-2015 14:48:46.009 -0800 INFO  ExecProcessor -     interval: 0 ms
01-14-2015 14:48:46.598 -0800 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_aws/bin/aws_cloudtrail.py" ERRORs3Bucket
0 Karma
1 Solution

bendter
Explorer

Okay, I figured out the issue. It was operator error.

We had configured SNS on the cloudtrail bucket NOT on cloudtrail itself. Everything in AWS appeared working since we were getting message in the SQS queue as files were added from cloudtrail.

I removed the SNS notification from the S3 bucket and configured cloudtrail to post to the SNS. I also removed and recreated the SQS queue to get notifications from the new SNS topic.

Everything is now working as it should.

Thanks to everyone who looked into this for us.

View solution in original post

bendter
Explorer

Okay, I figured out the issue. It was operator error.

We had configured SNS on the cloudtrail bucket NOT on cloudtrail itself. Everything in AWS appeared working since we were getting message in the SQS queue as files were added from cloudtrail.

I removed the SNS notification from the S3 bucket and configured cloudtrail to post to the SNS. I also removed and recreated the SQS queue to get notifications from the new SNS topic.

Everything is now working as it should.

Thanks to everyone who looked into this for us.

View solution in original post

jcoates_splunk
Splunk Employee
Splunk Employee

great! We're working on improving our troubleshooting guidance, but sadly I'm finding that a lot of the time it comes back to "nuke the AWS config and start over".

0 Karma

bendter
Explorer

The error message suggests a permission issue however I can log into AWS as the splunk user and download files from the cloudtrail S3 bucket manually. Originally I suspected our splunk credentials to AWS weren't working but I can use the same splunk AWS user to import data from cloudwatch.

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

hi, that typically indicates that it doesn't have permission to download the record from S3.

0 Karma

bendter
Explorer

I have further debugged this issue but I am still stumped. Please note I am NOT a JSON programmer.

The issue appearss to be caused by the the SQS message received by Splunk. Here is the line of code that generates the KeyError from aws_cloudtrail.py:

bucket_name = message['s3Bucket']

The key "S3Bucket" does not appear to exist. I dumped the SQS Messsage by adding this line to aws_cloudtrail.py:

logger.log(logging.INFO, "**** message: %s",json.dumps(message))

Here is the output (with some information obfuscated):

2015-01-24 11:06:11,299 INFO pid=9081 tid=MainThread file=aws_cloudtrail.py:process_notifications:295 | **** message: {"Records": [{"requestParameters": {"sourceIPAddress": "0.0.0.0"}, "userIdentity": {"principalId": "AWS:ARxxxxxxxxxxxxxxxxxxxx:i-69999999"}, "eventVersion": "2.0", "s3": {"bucket": {"ownerIdentity": {"principalId": "AAAAAAAAAAAAAA"}, "name": "cloudtrail", "arn": "arn:aws:s3:::cloudtrail"}, "s3SchemaVersion": "1.0", "object": {"eTag": "axxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "size": 8450, "key": "Folder1/Folder2/Folder3/us-west-2/2015/01/21/688888888888_CloudTrail_us-west-2_20150121T1000Z_XXXXXXXXXXXXXX.json.gz"}, "configurationId": "Splunk"}, "eventSource": "aws:s3", "responseElements": {"x-amz-id-2": "+XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=", "x-amz-request-id": "AAAAAAAAAAAAAAAA"}, "eventTime": "2015-01-21T10:00:03.710Z", "awsRegion": "us-west-2", "eventName": "ObjectCreated:Put"}]}

(continued in next comment)

bendter
Explorer

I also added this statement to see what "keys" were available:

logger.log(logging.INFO, "**** keys: %s",message.keys())

The output is:

2015-01-24 11:06:11,299 INFO pid=9081 tid=MainThread
file=aws_cloudtrail.py:process_notifications:297 | **** keys: [u'Records']

It looks to me like the only key is "Records" and all other keys are invalid. But since I don't understand JSON I don't really have much of an idea as to why the code is failing.

In another post someone said to try this in the code:

#message = json.loads(envelope['Message'])
message = envelope

I tried that and it didn't work. It gave the same KeyError message. Based on the apps directory in the post it looks like that was based on an earlier version of the AWS splunk add-on.

Any help would be much appreciated!

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

Can you try version 1.0.1 as a troubleshooting measure? it has some smarter parsing for invalid messages in an SQS queue, which is what I suspect you're seeing.

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.