Getting Data In

How to filter out debug logs except 3 different log types?

gyarici
Path Finder

Hi All,

I have some log data that includes INFO, WARN, ERROR and DEBUG levels.

I would like to index INFO, WARN, ERROR log messages and also three different log types in DEBUG.

I am doing DEBUG logs filtering using transforms.conf and props.conf to block specific logs, but as it is debug mode, there are many different types of logs and It is difficult to assign rules per each.

Is there anyway that I can block all kinds of DEBUG logs and only allow specific types which are 3 different logs only?

PS:I tried to use [setparsing] in transforms.conf but, it's not working.

transforms.conf
    [setnull]
    REGEX = DEBUG\w+
    DEST_KEY = queue
    FORMAT = nullQueue
    [setparsing]
    REGEX = DEBUG\s\[\w+]\s\[\w+]\s\ABC
    DEST_KEY = queue
    FORMAT = indexQueue

props.conf
[ABC]
TRANSFORMS-null = setnull

Thanks

Gokhan

0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

Your approach works if you add setparsing to your props.conf:

[ABC]
TRANSFORMS-null = setnull,setparsing

View solution in original post

dwaddle
SplunkTrust
SplunkTrust

Either of the answers below should work. The paste you have above looks like it came directly from the Splunk docs at http://docs.splunk.com/Documentation/Splunk/6.2.3/Forwarding/Routeandfilterdatad#Keep_specific_event... .. and there, they specifically demonstrate using both the setnull and setparsing stanzas.

In @woodcock's example, he uses a PCRE "negative lookahead assertion" in order to say "DEBUG, except when followed by.." See http://www.regular-expressions.info/lookaround.html for more information on how lookahead / lookbehind assertions work.

In @martin_mueller's example, he follows the documentation exactly and uses two different TRANSFORMS items that "fire" in a specific order. So, the first one (setnull) takes all items matching its REGEX and sets them to be sent to the nullQueue. The second one, (setparsing) takes all items matching its REGEX and sets them to be sent to the indexQueue. BOTH of these rules "fire" during processing, and the LAST one to match your data is the one that "wins"

You need to understand why these two different approaches exist, and why you might use one versus the other. For most humans who are not regex masters, the second approach by @martin_mueller is much easier to understand. And, there are some cases of complex rules where a lookahead assertion may not work for you. But, if you are a regex master, and an assertion does work for the situation you have, it is a perfectly good approach.

martin_mueller
SplunkTrust
SplunkTrust

Your approach works if you add setparsing to your props.conf:

[ABC]
TRANSFORMS-null = setnull,setparsing

gyarici
Path Finder

I think it was the fastest way according to my configuration. My mistake was to forget adding setparsing into my source-type in props.conf.

Here is my new configuration which is working perfectly.

transforms.conf
     [setnull]
     REGEX = DEBUG\w+
     DEST_KEY = queue
     FORMAT = nullQueue
     [setparsing]
     REGEX = DEBUG\s\[\w+]\s\[\w+]\s\ABC
     DEST_KEY = queue
     FORMAT = indexQueue

props.conf 
 [ABC]
 TRANSFORMS-null = setnull,setparsing

Thanks a lot for all of you.

Gokhan

woodcock
Esteemed Legend

Invert your RegEx and send everything that does not match the 3 keepers to the nullqueue; try this:

 [setnull]
 REGEX = DEBUG(?!\s\[\w+]\s\[\w+]\s\ABC)
 DEST_KEY = queue
 FORMAT = nullQueue

props.conf

[ABC]
TRANSFORMS-null = setnull

gyarici
Path Finder

Yes, this logic is also working ok too.

Thanks

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...