topic Re: Filtering the log using REGEX in Getting Data In

Filtering the log using REGEX

keshab — Wed, 02 Nov 2011 15:34:33 GMT

I have following log. What will be the REGEX to index log containing line the line 'tomcat' trying to restart and send rest to nullQueue??

[CDT Oct 27 09:31:44] info : 'tomcat' trying to restart
[CDT Oct 27 09:31:44] info : 'tomcat' start: /sbin/service
[CDT Oct 27 09:33:44] info : 'tomcat' process is running with pid 20021
[CDT Oct 27 12:13:44] error : 'tomcat' process PID changed to 6908
[CDT Oct 27 12:15:44] info : 'tomcat' PID has not changed
[CDT Oct 27 14:30:04] info : unmonitor service 'tomcat' on user request
[CDT Oct 27 14:30:04] info : monit daemon at 13351 awakened
[CDT Oct 27 14:30:04] info : Awakened by User defined signal 1
[CDT Oct 27 14:30:15] info : restart service 'tomcat' on user request
[CDT Oct 27 14:30:15] info : monit daemon at 13351 awakened
[CDT Oct 27 14:30:15] info : Awakened by User defined signal 1
[CDT Oct 27 14:30:15] info : 'tomcat' trying to restart
[CDT Oct 27 14:30:15] info : 'tomcat' stop: /sbin/service
[CDT Oct 27 14:30:43] info : 'tomcat' start: /sbin/service
[CDT Oct 27 14:31:43] error : 'tomcat' failed to start
[CDT Oct 27 14:32:43] error : 'tomcat' process is not running
[CDT Oct 27 14:32:43] info : 'tomcat' trying to restart
[CDT Oct 27 14:32:43] info : 'tomcat' start: /sbin/service
[CDT Oct 27 14:32:44] info : 'tomcat' started

Re: Filtering the log using REGEX

Ayn — Wed, 02 Nov 2011 15:41:51 GMT

First define a nullqueue transform, then another one to override it in the specific case where the event contains "'tomcat' trying to restart". Have a look at this topic in the docs: http://docs.splunk.com/Documentation/Splunk/4.1.5/Admin/Routeandfilterdata#Keep_specific_events_and_discard_the_rest

This should work:

props.conf:

[yoursourcetype]
TRANSFORMS-set= defaultnull,tomcatrestart

transforms.conf:

[defaultnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[tomcatrestart]
REGEX = 'tomcat' trying to restart
DEST_KEY = queue
FORMAT = indexQueue

Re: Filtering the log using REGEX

tgow — Wed, 02 Nov 2011 19:04:18 GMT

According to the docs if you want data to be sent to the null queue than you have to call your stanza in the transforms.conf setnull, ie:

props.conf

[yoursourcetype]
TRANSFORMS-set = setnull, setparsing

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \'tomcat\' trying to restart
DEST_KEY = queue
FORMAT = indexQueue

Re: Filtering the log using REGEX

Ayn — Wed, 02 Nov 2011 19:46:12 GMT

The transform doesn't have to be called "setnull", it's just what's used in the examples. You can call the transform anything, the only thing that matters is that it has any name so that there is something to refer to it by in props.conf.