I have some log data that includes INFO, WARN, ERROR and DEBUG levels.
I would like to index INFO, WARN, ERROR log messages and also three different log types in DEBUG.
I am doing DEBUG logs filtering using transforms.conf and props.conf to block specific logs, but as it is debug mode, there are many different types of logs and It is difficult to assign rules per each.
Is there anyway that I can block all kinds of DEBUG logs and only allow specific types which are 3 different logs only?
PS:I tried to use [setparsing] in transforms.conf but, it's not working.
transforms.conf [setnull] REGEX = DEBUG\w+ DEST_KEY = queue FORMAT = nullQueue [setparsing] REGEX = DEBUG\s\[\w+]\s\[\w+]\s\ABC DEST_KEY = queue FORMAT = indexQueue props.conf [ABC] TRANSFORMS-null = setnull
Invert your RegEx and send everything that does not match the 3 keepers to the
nullqueue; try this:
[setnull] REGEX = DEBUG(?!\s\[\w+]\s\[\w+]\s\ABC) DEST_KEY = queue FORMAT = nullQueue
[ABC] TRANSFORMS-null = setnull
Your approach works if you add
setparsing to your props.conf:
[ABC] TRANSFORMS-null = setnull,setparsing
I think it was the fastest way according to my configuration. My mistake was to forget adding setparsing into my source-type in props.conf.
Here is my new configuration which is working perfectly.
transforms.conf [setnull] REGEX = DEBUG\w+ DEST_KEY = queue FORMAT = nullQueue [setparsing] REGEX = DEBUG\s\[\w+]\s\[\w+]\s\ABC DEST_KEY = queue FORMAT = indexQueue props.conf [ABC] TRANSFORMS-null = setnull,setparsing
Thanks a lot for all of you.
Either of the answers below should work. The paste you have above looks like it came directly from the Splunk docs at http://docs.splunk.com/Documentation/Splunk/6.2.3/Forwarding/Routeandfilterdatad#Keep_specific_event... .. and there, they specifically demonstrate using both the
In @woodcock's example, he uses a PCRE "negative lookahead assertion" in order to say "DEBUG, except when followed by.." See http://www.regular-expressions.info/lookaround.html for more information on how lookahead / lookbehind assertions work.
In @martin_mueller's example, he follows the documentation exactly and uses two different
TRANSFORMS items that "fire" in a specific order. So, the first one (
setnull) takes all items matching its REGEX and sets them to be sent to the
nullQueue. The second one, (
setparsing) takes all items matching its REGEX and sets them to be sent to the
indexQueue. BOTH of these rules "fire" during processing, and the LAST one to match your data is the one that "wins"
You need to understand why these two different approaches exist, and why you might use one versus the other. For most humans who are not regex masters, the second approach by @martin_mueller is much easier to understand. And, there are some cases of complex rules where a lookahead assertion may not work for you. But, if you are a regex master, and an assertion does work for the situation you have, it is a perfectly good approach.