AWS Add on unable to parse CloudTrail data

[kkossery]

I have an issue where I get the error,

DEBUG pid=11513 tid=MainThread file=aws_cloudtrail.py:stream_events:210 | Connect to S3 & Sqs sucessfully
 2015-02-12 19:23:56,799 CRITICAL pid=11513 tid=MainThread file=aws_cloudtrail.py:stream_events:286 | Outer catchall: TypeError: 'int'            object has no attribute '__getitem__'
 2015-02-12 19:23:56,799 INFO pid=11513 tid=MainThread file=aws_cloudtrail.py:<module>:419 | EXITED: 1

And on Splunkd.log I see a generic error,

02-12-2015 19:36:27.297 +0000 ERROR ExecProcessor - message from "python /splunk/etc/apps/Splunk_TA_aws/bin/aws_cloudtrail.py" ERROR'int' object has no attribute '__getitem__'

It looks like Splunk is getting the logs to its indexes from AWS as I do a search on Splunk I can see JSON format logs there but the AWS add on is unable to parse the data and generate meaningful reports spewing out the above errors.
Can you guys help? I'm using the latest version of Splunk on Amazon Linux if that helps.

[kkossery · Accepted solution]

After spending time on the Splunk forums and finding this link, this has been resolved. I had to create an index named aws-cloudtrail manually and load the data in it.

http://answers.splunk.com/answers/205803/aws-cloudtrail-data-not-shown-in-the-dashboard-und.html

[kkossery]

After re-doing my Splunk Install and SQS, SNS, CloudTrail setup, I see some improvement. On aws_CloudTrail Log,

2015-02-13 11:03:44,241 INFO pid=25836 tid=MainThread file=aws_cloudtrail.py:process_notifications:356 |    processing 4 records in s3:trailertruck/AWSLogs/2xxxxxxx/CloudTrail/us-east-1/2015/02/13/xxxxxxxx_CloudTrail_us-east-1_20150213T1605Z_0YodSeqjgEBI0nqU.json.gz
2015-02-13 11:03:44,241 DEBUG pid=25836 tid=MainThread file=aws_cloudtrail.py:process_notifications:369 | writing event DescribeLoadBalancers with timestamp 2015-02-13T16:00:02Z
2015-02-13 11:03:44,242 DEBUG pid=25836 tid=MainThread file=aws_cloudtrail.py:process_notifications:369 | writing event CreateKeyPair with timestamp 2015-02-13T15:59:25Z
2015-02-13 11:03:44,243 DEBUG pid=25836 tid=MainThread file=aws_cloudtrail.py:process_notifications:369 | writing event DescribeAlarms with timestamp 2015-02-13T15:59:11Z
2015-02-13 11:03:44,243 DEBUG pid=25836 tid=MainThread file=aws_cloudtrail.py:process_notifications:369 | writing event DeleteKeyPair with timestamp 2015-02-13T15:59:49Z
2015-02-13 11:03:44,244 INFO pid=25836 tid=MainThread file=aws_cloudtrail.py:process_notifications:393 | fetched 4 records, wrote 4, discarded 0, redirected 0 from s3:trailertruck/AWSLogs/2xxxxxxx/CloudTrail/us-east-1/2015/02/13/2xxxxxxxxx_CloudTrail_us-east-1_20150213T1605Z_0YodSeqjgEBI0nqU.json.gz
2015-02-13 11:03:44,256 INFO pid=25836 tid=MainThread file=aws_cloudtrail.py:stream_events:283 | 1 completed, 0 failed while processing a notification batch of 1 [0 errors deleting 1 notifications]  Elapsed: 0.077s

However, on splunkd.log

02-13-2015 10:55:36.381 -0500 WARN  SearchOperator:inputcsv - Encountered 1 'inconsistent number of column' errors while reading input.
    02-13-2015 10:55:36.791 -0500 WARN  SearchOperator:inputcsv - Encountered 1 'inconsistent number of column' errors while reading input.
    02-13-2015 10:55:36.809 -0500 WARN  SearchOperator:inputcsv - Encountered 1 'inconsistent number of column' errors while reading input.

I'm not sure what to make of it but I'm going to dig deeper and see if I can come up with something else.

[kkossery]

Thank you jcoates_splunk for your response to another thread. He responded by,
"every time i've seen that sort of error message it's meant that the add-on is being directed to gather "cloudtrail" data from a bucket that actually contains something else."

Here is my response - The cloudtrail folder was created specifically for this purpose and do not have any other data. Is there any other suggestions that I need to try out? Thank you again.