Want to forward selective portion of the log file by forwarder to indexer

[somnathnag]

Hello :
What I have to set in inputs.conf or outputs.conf of the forwarder so that I can send selective portion of the application log file to indexer. At present our setup forwards the whole log file - which we don't need and this is hurting the network performance also.
Thanks
Somnath

[lguinn2]

Example: Assume that the data you want to filter is in a file named big.log. (Note that in real life, you would have to provide the full path.)

You want to eliminate any lines that have the word INFO or WARN.

In the configuration file $SPLUNK_HOME\etc\system\local\props.conf

[source::big.log]
TRANSFORMS-t1 = filterEvents

In the configuration file $SPLUNK_HOME\etc\system\local\transforms.conf

[filterEvents]
REGEX = (?:INFO|WARN)
DEST_KEY = queue
FORMAT = nullQueue

[lguinn2 · Accepted solution]

Take a look at the following in the manual

Perform selective indexing and forwarding

You can filter the data in two different places:

• Option 1: Filter on the indexer(s). This is the recommendation when you are filtering less than 50% of the data. Why? Because by filtering on the indexer(s), you can use the Universal Forwarder on the production systems.
• Option 2: Filter on the forwarders. This requires that you use the heavy forwarder instead of the Universal Forwarder, and it will take some processing cycles on the production systems. Still, this may be the best choice if you are seeing network performance issues, or if you are filtering out more than 50% of the data.

The syntax and configuration files are exactly the same in either case - it is simply a matter of where you put them.

[somnathnag]

Thanks a lot.
Probably we would go with the heavy forwarder. I looked into your attached document - couldn't figure out the configuration for selective event forwarding. Can you please provide a sample configuration file ?