All Apps and Add-ons

AWS Add on unable to parse CloudTrail data

kkossery
Communicator

I have an issue where I get the error,

DEBUG pid=11513 tid=MainThread file=aws_cloudtrail.py:stream_events:210 | Connect to S3 & Sqs sucessfully
 2015-02-12 19:23:56,799 CRITICAL pid=11513 tid=MainThread file=aws_cloudtrail.py:stream_events:286 | Outer catchall: TypeError: 'int'            object has no attribute '__getitem__'
 2015-02-12 19:23:56,799 INFO pid=11513 tid=MainThread file=aws_cloudtrail.py:<module>:419 | EXITED: 1

And on Splunkd.log I see a generic error,

02-12-2015 19:36:27.297 +0000 ERROR ExecProcessor - message from "python /splunk/etc/apps/Splunk_TA_aws/bin/aws_cloudtrail.py" ERROR'int' object has no attribute '__getitem__'

It looks like Splunk is getting the logs to its indexes from AWS as I do a search on Splunk I can see JSON format logs there but the AWS add on is unable to parse the data and generate meaningful reports spewing out the above errors.
Can you guys help? I'm using the latest version of Splunk on Amazon Linux if that helps.

0 Karma
1 Solution

kkossery
Communicator

After spending time on the Splunk forums and finding this link, this has been resolved. I had to create an index named aws-cloudtrail manually and load the data in it.

http://answers.splunk.com/answers/205803/aws-cloudtrail-data-not-shown-in-the-dashboard-und.html

View solution in original post

kkossery
Communicator

After spending time on the Splunk forums and finding this link, this has been resolved. I had to create an index named aws-cloudtrail manually and load the data in it.

http://answers.splunk.com/answers/205803/aws-cloudtrail-data-not-shown-in-the-dashboard-und.html

kkossery
Communicator

After re-doing my Splunk Install and SQS, SNS, CloudTrail setup, I see some improvement. On aws_CloudTrail Log,

2015-02-13 11:03:44,241 INFO pid=25836 tid=MainThread file=aws_cloudtrail.py:process_notifications:356 |    processing 4 records in s3:trailertruck/AWSLogs/2xxxxxxx/CloudTrail/us-east-1/2015/02/13/xxxxxxxx_CloudTrail_us-east-1_20150213T1605Z_0YodSeqjgEBI0nqU.json.gz
2015-02-13 11:03:44,241 DEBUG pid=25836 tid=MainThread file=aws_cloudtrail.py:process_notifications:369 | writing event DescribeLoadBalancers with timestamp 2015-02-13T16:00:02Z
2015-02-13 11:03:44,242 DEBUG pid=25836 tid=MainThread file=aws_cloudtrail.py:process_notifications:369 | writing event CreateKeyPair with timestamp 2015-02-13T15:59:25Z
2015-02-13 11:03:44,243 DEBUG pid=25836 tid=MainThread file=aws_cloudtrail.py:process_notifications:369 | writing event DescribeAlarms with timestamp 2015-02-13T15:59:11Z
2015-02-13 11:03:44,243 DEBUG pid=25836 tid=MainThread file=aws_cloudtrail.py:process_notifications:369 | writing event DeleteKeyPair with timestamp 2015-02-13T15:59:49Z
2015-02-13 11:03:44,244 INFO pid=25836 tid=MainThread file=aws_cloudtrail.py:process_notifications:393 | fetched 4 records, wrote 4, discarded 0, redirected 0 from s3:trailertruck/AWSLogs/2xxxxxxx/CloudTrail/us-east-1/2015/02/13/2xxxxxxxxx_CloudTrail_us-east-1_20150213T1605Z_0YodSeqjgEBI0nqU.json.gz
2015-02-13 11:03:44,256 INFO pid=25836 tid=MainThread file=aws_cloudtrail.py:stream_events:283 | 1 completed, 0 failed while processing a notification batch of 1 [0 errors deleting 1 notifications]  Elapsed: 0.077s

However, on splunkd.log

02-13-2015 10:55:36.381 -0500 WARN  SearchOperator:inputcsv - Encountered 1 'inconsistent number of column' errors while reading input.
    02-13-2015 10:55:36.791 -0500 WARN  SearchOperator:inputcsv - Encountered 1 'inconsistent number of column' errors while reading input.
    02-13-2015 10:55:36.809 -0500 WARN  SearchOperator:inputcsv - Encountered 1 'inconsistent number of column' errors while reading input.

I'm not sure what to make of it but I'm going to dig deeper and see if I can come up with something else.

0 Karma

kkossery
Communicator

Thank you jcoates_splunk for your response to another thread. He responded by,
"every time i've seen that sort of error message it's meant that the add-on is being directed to gather "cloudtrail" data from a bucket that actually contains something else."

Here is my response - The cloudtrail folder was created specifically for this purpose and do not have any other data. Is there any other suggestions that I need to try out? Thank you again.

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...