After re-doing my Splunk Install and SQS, SNS, CloudTrail setup, I see some improvement. On aws_CloudTrail Log,
2015-02-13 11:03:44,241 INFO pid=25836 tid=MainThread file=aws_cloudtrail.py:process_notifications:356 | processing 4 records in s3:trailertruck/AWSLogs/2xxxxxxx/CloudTrail/us-east-1/2015/02/13/xxxxxxxx_CloudTrail_us-east-1_20150213T1605Z_0YodSeqjgEBI0nqU.json.gz
2015-02-13 11:03:44,241 DEBUG pid=25836 tid=MainThread file=aws_cloudtrail.py:process_notifications:369 | writing event DescribeLoadBalancers with timestamp 2015-02-13T16:00:02Z
2015-02-13 11:03:44,242 DEBUG pid=25836 tid=MainThread file=aws_cloudtrail.py:process_notifications:369 | writing event CreateKeyPair with timestamp 2015-02-13T15:59:25Z
2015-02-13 11:03:44,243 DEBUG pid=25836 tid=MainThread file=aws_cloudtrail.py:process_notifications:369 | writing event DescribeAlarms with timestamp 2015-02-13T15:59:11Z
2015-02-13 11:03:44,243 DEBUG pid=25836 tid=MainThread file=aws_cloudtrail.py:process_notifications:369 | writing event DeleteKeyPair with timestamp 2015-02-13T15:59:49Z
2015-02-13 11:03:44,244 INFO pid=25836 tid=MainThread file=aws_cloudtrail.py:process_notifications:393 | fetched 4 records, wrote 4, discarded 0, redirected 0 from s3:trailertruck/AWSLogs/2xxxxxxx/CloudTrail/us-east-1/2015/02/13/2xxxxxxxxx_CloudTrail_us-east-1_20150213T1605Z_0YodSeqjgEBI0nqU.json.gz
2015-02-13 11:03:44,256 INFO pid=25836 tid=MainThread file=aws_cloudtrail.py:stream_events:283 | 1 completed, 0 failed while processing a notification batch of 1 [0 errors deleting 1 notifications] Elapsed: 0.077s
However, on splunkd.log
02-13-2015 10:55:36.381 -0500 WARN SearchOperator:inputcsv - Encountered 1 'inconsistent number of column' errors while reading input.
02-13-2015 10:55:36.791 -0500 WARN SearchOperator:inputcsv - Encountered 1 'inconsistent number of column' errors while reading input.
02-13-2015 10:55:36.809 -0500 WARN SearchOperator:inputcsv - Encountered 1 'inconsistent number of column' errors while reading input.
I'm not sure what to make of it but I'm going to dig deeper and see if I can come up with something else.
... View more