How to limit the amount of data that a splunk universal forwarder sends to the Splunk server for processing?

[vvmmvvmm]

Hi all

I am using Splunk Enterprise for security...

But I have a lot of extraneous data in Splunk at the moment. Looking through the dashboards I'm finding a lot of performance and operational status data.

The problem is that my splunk license allows me to analyze 2gb of data in a 24 hour period. I would say that at the moment 70% of the data that goes through the system is not security related and the system was procured as a security monitoring system.

I would like to find a way to reduce the mount of the data that the "forwarders" send back to the Splunk back end for processing. i.e. exclude all of the performance and operational data from the analysis.

My intention is to use that freed up bandwidth to push the Sophos Anti Virus and Firewall logs I have, to splunk instead of server performance data.

Is this possible, and can anybody provide me with details on how to do this? I would really really appreciate your help! I have searched online for an answer but so far I can't find anything, but if you know of a page where I can find the information I need, please do send me a link 🙂

Kind Regards

Vera

[ChrisG]

There is a doc topic about routing and filtering data, which includes information about universal forwarders as well as heavy forwarders: Route and filter data in the Forwarding Data manual.

Another thing you might want to look at is David Paper's presentation from the 2014 user conference, "Getting the Most Out of Your Splunk License: Keeping the Junk Out of Splunk."

• Slides
• Recording

[vvmmvvmm]

Thank you Chris, and the presentation was really helpful 🙂

[ddrillic · Accepted solution]

Basically, you have two places to control the data flow -
1) At the forwarder level

http://docs.splunk.com/Documentation/Splunk/6.4.2/Admin/Inputsconf
is a great place to start especially at the monitor options.

2) At the parsing queue on the indexer

If you block data in one of these two points, the data won't count against your license.

[vvmmvvmm]

Thank you very much ddrillic 🙂