Getting Data In

How to configure selective data indexing ?

somnathnag
Engager

Our configuration has universal forwarder - so the whole log file is being forwarded to the indexer. I know there is lot of data/information/warning in the log file which is not required for monitoring. How can I ignore those data so that indexer don't have to index so much (not needed) data ? I understand there is some configuration for that - but couldn't figure out from the deployment document of splunk. Or am I looking at the wrong document ?

Tags (2)
0 Karma
1 Solution

kristian_kolb
Ultra Champion

You do that through props.conf and transforms.conf on the indexer, since that is where the parsing takes place. If you hava a full/heavy forwarder, you can do the operation there.

For instance, if your special log file contains a lot of events containing WARNING, and you don't want to index them, your config should look something like;

In props.conf

[source::/var/your_special.log]
TRANSFORMS-set= setnull

In transforms.conf

[setnull]
REGEX=WARNING
DEST_KEY=queue
FORMAT=nullQueue

Also see;

http://www.splunk.com/wiki/Where_do_I_configure_my_Splunk_settings

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf

Hope this helps,

Kristian

View solution in original post

0 Karma

kristian_kolb
Ultra Champion

You do that through props.conf and transforms.conf on the indexer, since that is where the parsing takes place. If you hava a full/heavy forwarder, you can do the operation there.

For instance, if your special log file contains a lot of events containing WARNING, and you don't want to index them, your config should look something like;

In props.conf

[source::/var/your_special.log]
TRANSFORMS-set= setnull

In transforms.conf

[setnull]
REGEX=WARNING
DEST_KEY=queue
FORMAT=nullQueue

Also see;

http://www.splunk.com/wiki/Where_do_I_configure_my_Splunk_settings

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf

Hope this helps,

Kristian

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Accelerate Splunk App Development with AI

  Learn how to automate the repetitive parts of your build cycle.   Watch On-Demand   Replay the Accelerate ...

Splunk Agent Observability: Measuring AI Value with Splunk Tokenomics on AWS

The AI landscape is moving faster than ever, and our latest 2026 State of Observability report provides the ...

Enterprise Security Watch Party: What’s New in 8.8 - Recap

Splunk Community HQ hosted a Splunk Enterprise Security Watch Party: What’s New In ES 8.8 last Thursday. I had ...