Unable to collect data from AWS SQS with Splunk Add-On for AWS

[david_emind]

Hi All, Would appreciate some suggestions to a solution. Thanks!

I am unable to collect any data from AWS SQS. Brand new AWS Linux OS(yum update) with Splunk Enterprise 6.2.1 and Add-on(Version 1.0.1) and App(Version 3.0) for AWS installed. splunk user in IAM has full permissions to SQS and S3. SQS subscribed to SNS topic and is showing messages in the queue. In addition, there is a index that was manually created called aws-cloudtrail for which is required by SplunkAppforAWS.

**This is the output of my log file aws_cloudtrail.log

2015-01-03 10:09:28,865 INFO pid=30098 tid=MainThread file=aws_cloudtrail.py::413 | STARTED:
2015-01-03 10:09:28,865 DEBUG pid=30098 tid=MainThread file=aws_cloudtrail.py:stream_events:174 | Start streaming.
2015-01-03 10:09:28,865 DEBUG pid=30098 tid=MainThread file=aws_cloudtrail.py:stream_events:192 | blacklist regex for eventNames is None
2015-01-03 10:09:28,866 INFO pid=30098 tid=MainThread file=aws_cloudtrail.py:get_access_key_pwd_real:105 | get account name: splunk
2015-01-03 10:09:28,887 DEBUG pid=30098 tid=MainThread file=aws_cloudtrail.py:stream_events:206 | Connect to S3 & Sqs sucessfully
2015-01-03 10:09:28,981 CRITICAL pid=30098 tid=MainThread file=aws_cloudtrail.py:stream_events:282 | Outer catchall: ParseError: no element found: line 1, column 0
2015-01-03 10:09:28,982 INFO pid=30098 tid=MainThread file=aws_cloudtrail.py::415 | EXITED: 1

**I'm also seeing messages like this in the splunkd.log.
01-03-2015 09:17:11.556 +0000 WARN SearchOperator:inputcsv - Encountered 1 'inconsistent number of column' errors while reading input.
01-03-2015 09:18:28.428 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_aws/bin/aws_cloudtrail.py" ERRORno element found: line 1, column 0

Any clues why?

[jcoates_splunk]

That usually indicates that it's pulling a message from SQS that isn't from CloudTrail. As of the latest 1.0.x it should write the message to a log and delete it, but if it doesn't have permission to delete it might get stuck on the same message.

[david_emind]

This is a fresh install and I didn't expect this to happen. What do you think the solution could be? I can try it out and get back to you.
Thanks!

[kkossery]

Hello David - Were you able to find a solution to this? We see the exact problem you described.
I don't have any other SQS queue or SNS topic besides the one for CloudTrail.

[jcoates_splunk]

with the current 1.1.0 version of the Add-on, it should log that it's seeing messages that aren't CloudTrail format and delete them from the queue so that it can proceed with the CloudTrail data.

[kkossery]

Thanks! We were able to make it work earlier.

[jcoates_splunk]

Can you try deleting the SQS message which isn't from CloudTrail?