Filtering application log

[keshab]

I am new to this Splunk...I have universal forwarder installed on my two linux machine and successfully forwarding apache, tomcat logs to another linux machine which has Splunk server(indexer) using tcp port. Now I want to filter those logs based on REGEX. I was just wondering which files and which section of the files should I make the changes on universal forwarder and splunk index??

[jasonchangbompa]

Hi, I have an open question about splunk "log filtering" performance.

Is there any document or technical report mentioning about performance metrics which clearly shows case-by-case (e.g. depending on number of cores, etc) filtering performance?

Nothing comes out of my mind at this point regarding this performance issue, yet.
But I guess you know what I'm talking about...

Best Wishes, Sung Jin

[kdenton]

You could also install the heavy forwarder which is basically a full install of Splunk with some features disabled to help reduce the foot print.

Useing the heavy forwarder you can filter based on regex.

[_d_]

Keshab,

A universal forwarder is not able to do any filtering based on regex (with the exception of having regex based whitelists in your inputs.conf stanza so that you selectively bring files in to your Splunk indexer) but an indexer will.

There are two main files that you can use to achieve what you're looking for. Create these file in $SPLUNK_HOME/etc/system/local and modify them accordingly.

1. props.conf :: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf
   
2. transforms.conf :: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf

The links above cover all file options/parameters and also come with a few examples.
A few more resources:

field extractions: http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Createandmaintainsearch-timefieldextrac...

route and filter logs: http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Routeandfilterdatad

Hope this helps.

> please upvote and accept answer if you find it useful - thanks!