Splunk Add-on for Amazon Web Services 1.1.1: Why are some Cloudtrail logs not being collected?

[stevepraz]

We recently configured the Splunk Add-on for Amazon Web Services to pull Cloudtrail data from our AWS account. We are running Splunk Enterprise 6.2.2 and version 1.1.1 of the add-on for AWS.

We are experiencing a weird behavior where some entries of a particular Cloudtrail log file are collected and others are not. In one source log file, 5 of 17 entries were captured. There doesn't seem to be an obvious rhyme or reason regarding the items that are missed. In this particular example, the 5 that were captured are throughout the file, not just in the beginning or end.

I don't think it's a permissions issues since they are all in the same file. I'm also not aware of any filtering setup on my input that would create this type of behavior.

[stevepraz · Accepted solution]

Found the issue thanks to some previous Splunk Answers and digging in the logs.

I noticed snippet of a message from the Splunk aws_cloudtrail.log:
"fetched 16 records, wrote 1, discarded 15, redirected 0 from s3"

Searching on that led me to this Splunk Answer:
link text

The gist being that there is a default property in the inputs.conf called exclude_describe_events that prevents the AWS add on from pulling down most "read-only" type events. It's exactly clear that this is happening if you configure the cloudtrail input via Splunk Web.

Once I changed that property to false, all the entries in the cloudtrail logs started coming through normally.