How to configure selective data indexing ?

[somnathnag]

Our configuration has universal forwarder - so the whole log file is being forwarded to the indexer. I know there is lot of data/information/warning in the log file which is not required for monitoring. How can I ignore those data so that indexer don't have to index so much (not needed) data ? I understand there is some configuration for that - but couldn't figure out from the deployment document of splunk. Or am I looking at the wrong document ?

[kristian_kolb · Accepted solution]

You do that through props.conf and transforms.conf on the indexer, since that is where the parsing takes place. If you hava a full/heavy forwarder, you can do the operation there.

For instance, if your special log file contains a lot of events containing WARNING, and you don't want to index them, your config should look something like;

In props.conf

[source::/var/your_special.log]
TRANSFORMS-set= setnull

In transforms.conf

[setnull]
REGEX=WARNING
DEST_KEY=queue
FORMAT=nullQueue

Also see;

http://www.splunk.com/wiki/Where_do_I_configure_my_Splunk_settings

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf

Hope this helps,

Kristian