Hello everyone. I am very new to Splunk and I am trying to filter logs before they reach the indexer. I literally hit the 500mb daily cap in 20 minutes, especially with security logs. Does anyone have a non vague guide/template of where to start? The most I have done was create the .confg files for props and transforms in C:\Program Files\Splunk\etc\system\local as the Splunk documentation mentioned. Thank you very much!
I would recommend installing the deployment monitor app and splunk on splunk app and watch the system for a day or so. Get an understanding of what volumes are for the different sourcetypes and hosts, and like you said, get together with folks and decide what data you want to keep and what data can be discarded or ignored all together.
Keep in mind, if the system is logging data, and the logs are being saved then there is no rush to index them because the saved log files can be indexed later if the need arrises. If you drop data prior to indexing, then it is harder to get that data back.
The latest 6.0.1 So my team is going to sit down and discuss what we are wanting to exclude. I guess I was just looking for a good reference of how or where to start. I still have a lot to go over, and am still very new.
If you want a "non vague" answer, we need more information! Splunk is very flexible, but that means that you have to be specific about what to include/exclude.
As Luke mentions, there is a easy way to exclude some data at the forwarder in Splunk 6, but not earlier versions.
In general, data is excluded when it is parsed - usually at the indexer. However, parsing occurs BEFORE the license meter, so data excluded at the indexer does not count against the 500MB.
If you really want to exclude the data at the forwarder, you will need to use a heavy forwarder, and place a copy of your props.conf
and transforms.conf
on the forwarder. The heavy forwarder will parse the data and then forward it. The universal forwarder cannot parse the data.
And as you are new to splunk, you could go for splunk 6 where it is having some ready made functionality for filtering.
_http://blogs.splunk.com/2013/10/14/windows-event-logs-in-splunk-6/
And if you would like to do it the conventional way
_http://answers.splunk.com/answers/29218/filtering-windows-event-logs
Which version of splunk are you using, and which EventCodes do you want to drop?
Are you using the deployment monitor app to evaluate the index volume per sourcetype?