Getting Data In

How to filter event logs before indexing

Szethius
Explorer

Hello everyone. I am very new to Splunk and I am trying to filter logs before they reach the indexer. I literally hit the 500mb daily cap in 20 minutes, especially with security logs. Does anyone have a non vague guide/template of where to start? The most I have done was create the .confg files for props and transforms in C:\Program Files\Splunk\etc\system\local as the Splunk documentation mentioned. Thank you very much!

0 Karma

lukejadamec
Super Champion

I would recommend installing the deployment monitor app and splunk on splunk app and watch the system for a day or so. Get an understanding of what volumes are for the different sourcetypes and hosts, and like you said, get together with folks and decide what data you want to keep and what data can be discarded or ignored all together.
Keep in mind, if the system is logging data, and the logs are being saved then there is no rush to index them because the saved log files can be indexed later if the need arrises. If you drop data prior to indexing, then it is harder to get that data back.

Szethius
Explorer

The latest 6.0.1 So my team is going to sit down and discuss what we are wanting to exclude. I guess I was just looking for a good reference of how or where to start. I still have a lot to go over, and am still very new.

0 Karma

lguinn2
Legend

If you want a "non vague" answer, we need more information! Splunk is very flexible, but that means that you have to be specific about what to include/exclude.

As Luke mentions, there is a easy way to exclude some data at the forwarder in Splunk 6, but not earlier versions.

In general, data is excluded when it is parsed - usually at the indexer. However, parsing occurs BEFORE the license meter, so data excluded at the indexer does not count against the 500MB.

If you really want to exclude the data at the forwarder, you will need to use a heavy forwarder, and place a copy of your props.conf and transforms.conf on the forwarder. The heavy forwarder will parse the data and then forward it. The universal forwarder cannot parse the data.

0 Karma

linu1988
Champion

And as you are new to splunk, you could go for splunk 6 where it is having some ready made functionality for filtering.

_http://blogs.splunk.com/2013/10/14/windows-event-logs-in-splunk-6/

And if you would like to do it the conventional way

_http://answers.splunk.com/answers/29218/filtering-windows-event-logs

0 Karma

lukejadamec
Super Champion

Which version of splunk are you using, and which EventCodes do you want to drop?
Are you using the deployment monitor app to evaluate the index volume per sourcetype?

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...