Getting Data In
Highlighted

How to filter out debug logs except 3 different log types?

Path Finder

Hi All,

I have some log data that includes INFO, WARN, ERROR and DEBUG levels.

I would like to index INFO, WARN, ERROR log messages and also three different log types in DEBUG.

I am doing DEBUG logs filtering using transforms.conf and props.conf to block specific logs, but as it is debug mode, there are many different types of logs and It is difficult to assign rules per each.

Is there anyway that I can block all kinds of DEBUG logs and only allow specific types which are 3 different logs only?

PS:I tried to use [setparsing] in transforms.conf but, it's not working.

transforms.conf
    [setnull]
    REGEX = DEBUG\w+
    DEST_KEY = queue
    FORMAT = nullQueue
    [setparsing]
    REGEX = DEBUG\s\[\w+]\s\[\w+]\s\ABC
    DEST_KEY = queue
    FORMAT = indexQueue

props.conf
[ABC]
TRANSFORMS-null = setnull

Thanks

Gokhan

0 Karma
Highlighted

Re: How to filter out debug logs except 3 different log types?

Esteemed Legend

Invert your RegEx and send everything that does not match the 3 keepers to the nullqueue; try this:

 [setnull]
 REGEX = DEBUG(?!\s\[\w+]\s\[\w+]\s\ABC)
 DEST_KEY = queue
 FORMAT = nullQueue

props.conf

[ABC]
TRANSFORMS-null = setnull
Highlighted

Re: How to filter out debug logs except 3 different log types?

Path Finder

Yes, this logic is also working ok too.

Thanks

0 Karma
Highlighted

Re: How to filter out debug logs except 3 different log types?

SplunkTrust
SplunkTrust

Your approach works if you add setparsing to your props.conf:

[ABC]
TRANSFORMS-null = setnull,setparsing

View solution in original post

Highlighted

Re: How to filter out debug logs except 3 different log types?

Path Finder

I think it was the fastest way according to my configuration. My mistake was to forget adding setparsing into my source-type in props.conf.

Here is my new configuration which is working perfectly.

transforms.conf
     [setnull]
     REGEX = DEBUG\w+
     DEST_KEY = queue
     FORMAT = nullQueue
     [setparsing]
     REGEX = DEBUG\s\[\w+]\s\[\w+]\s\ABC
     DEST_KEY = queue
     FORMAT = indexQueue

props.conf 
 [ABC]
 TRANSFORMS-null = setnull,setparsing

Thanks a lot for all of you.

Gokhan

Highlighted

Re: How to filter out debug logs except 3 different log types?

SplunkTrust
SplunkTrust

Either of the answers below should work. The paste you have above looks like it came directly from the Splunk docs at http://docs.splunk.com/Documentation/Splunk/6.2.3/Forwarding/Routeandfilterdatad#Keep_specific_event... .. and there, they specifically demonstrate using both the setnull and setparsing stanzas.

In @woodcock's example, he uses a PCRE "negative lookahead assertion" in order to say "DEBUG, except when followed by.." See http://www.regular-expressions.info/lookaround.html for more information on how lookahead / lookbehind assertions work.

In @martin_mueller's example, he follows the documentation exactly and uses two different TRANSFORMS items that "fire" in a specific order. So, the first one (setnull) takes all items matching its REGEX and sets them to be sent to the nullQueue. The second one, (setparsing) takes all items matching its REGEX and sets them to be sent to the indexQueue. BOTH of these rules "fire" during processing, and the LAST one to match your data is the one that "wins"

You need to understand why these two different approaches exist, and why you might use one versus the other. For most humans who are not regex masters, the second approach by @martin_mueller is much easier to understand. And, there are some cases of complex rules where a lookahead assertion may not work for you. But, if you are a regex master, and an assertion does work for the situation you have, it is a perfectly good approach.