Hi All,
I have some log data that includes INFO, WARN, ERROR and DEBUG levels.
I would like to index INFO, WARN, ERROR log messages and also three different log types in DEBUG.
I am doing DEBUG logs filtering using transforms.conf and props.conf to block specific logs, but as it is debug mode, there are many different types of logs and It is difficult to assign rules per each.
Is there anyway that I can block all kinds of DEBUG logs and only allow specific types which are 3 different logs only?
PS:I tried to use [setparsing] in transforms.conf but, it's not working.
transforms.conf
[setnull]
REGEX = DEBUG\w+
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = DEBUG\s\[\w+]\s\[\w+]\s\ABC
DEST_KEY = queue
FORMAT = indexQueue
props.conf
[ABC]
TRANSFORMS-null = setnull
Thanks
Gokhan
Your approach works if you add setparsing
to your props.conf:
[ABC]
TRANSFORMS-null = setnull,setparsing
Either of the answers below should work. The paste you have above looks like it came directly from the Splunk docs at http://docs.splunk.com/Documentation/Splunk/6.2.3/Forwarding/Routeandfilterdatad#Keep_specific_event... .. and there, they specifically demonstrate using both the setnull
and setparsing
stanzas.
In @woodcock's example, he uses a PCRE "negative lookahead assertion" in order to say "DEBUG, except when followed by.." See http://www.regular-expressions.info/lookaround.html for more information on how lookahead / lookbehind assertions work.
In @martin_mueller's example, he follows the documentation exactly and uses two different TRANSFORMS
items that "fire" in a specific order. So, the first one (setnull
) takes all items matching its REGEX and sets them to be sent to the nullQueue
. The second one, (setparsing
) takes all items matching its REGEX and sets them to be sent to the indexQueue
. BOTH of these rules "fire" during processing, and the LAST one to match your data is the one that "wins"
You need to understand why these two different approaches exist, and why you might use one versus the other. For most humans who are not regex masters, the second approach by @martin_mueller is much easier to understand. And, there are some cases of complex rules where a lookahead assertion may not work for you. But, if you are a regex master, and an assertion does work for the situation you have, it is a perfectly good approach.
Your approach works if you add setparsing
to your props.conf:
[ABC]
TRANSFORMS-null = setnull,setparsing
I think it was the fastest way according to my configuration. My mistake was to forget adding setparsing into my source-type in props.conf.
Here is my new configuration which is working perfectly.
transforms.conf
[setnull]
REGEX = DEBUG\w+
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = DEBUG\s\[\w+]\s\[\w+]\s\ABC
DEST_KEY = queue
FORMAT = indexQueue
props.conf
[ABC]
TRANSFORMS-null = setnull,setparsing
Thanks a lot for all of you.
Gokhan
Invert your RegEx and send everything that does not match the 3 keepers to the nullqueue
; try this:
[setnull]
REGEX = DEBUG(?!\s\[\w+]\s\[\w+]\s\ABC)
DEST_KEY = queue
FORMAT = nullQueue
props.conf
[ABC]
TRANSFORMS-null = setnull
Yes, this logic is also working ok too.
Thanks