Splunk App for AWS: How to prevent Splunk AWS API queries from consuming 80% of my daily license?

[chrisprangnell]

Before I added IAM logging for CloudTrail, my Splunk volume was at 150meg for the day.
After I added IAM logging for CloudTrail, my splunk volume moved to 700meg a day. A huge huge shift.

Most of which seems to be Splunk account instigated events querying the AWS API for information to populate the AWS app dashboards. AWS API describe events I'm guessing

How can we stop Splunk related activity being pulled and using 80% of my license?

[_d_ · Accepted solution]

Chris, note that for CloudTrail you can blacklist based on event names. E.g.: this blacklist regex will omit Describes, List and Get events:

blacklist = ^(?:Describe|List|Get)

See more here: http://docs.splunk.com/Documentation/AddOns/latest/AWS/CloudTrail

[chrisprangnell]

But how would that help me?

I only want to stop pulling data that has my splunk access code in it? I think that would filter all the data caused by getting the data if that makes sense.

[_d_]

Ah okay, in that case - to scope filtering on a specific string of the event you need to use props/transforms. There is a sample here: http://docs.splunk.com/Documentation/Splunk/6.4.1/Forwarding/Routeandfilterdatad#Discard_specific_ev...

[chrisprangnell]

The problem with this is that im still using my Splunk volume license to pull data i do not want.

[_d_]

Filtering data out before indexing does not impact license.

[chrisprangnell]

Brilliant thank you!

[chrisprangnell]

The different of which is 4-6gig a day compared to 120meg.