Why are /var/log/messages filter null queue not filtering?

[youngsuh]

Need regex & Null queue help to send events in /var/log/messages.

Here is regex101:

regex101: build, test, and debug regex  

 (IP & hostname randomized)

props.conf
[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing

transforms.conf
[setnull]
REGEX = \w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\w+\n
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\w{5}\d{4}\S\i.ab2.jone.com\s.+\n
DEST_KEY = queue
FORMAT = indexQueue

the regex not sending unwanted event in /var/log/message .  I am doing the on HF before UF.

[gcusello · Accepted solution]

Hi @youngsuh,

if you want to take from /var/log/messages only events containing "i.ab2.jone.com", you should try something like this:

props.conf
[source::/var/log/messages]
TRANSFORMS-set = setnull,setparsing

transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = i\.ab2\.jone\.com
DEST_KEY = queue
FORMAT = indexQueue

Ciao.

Giuseppe

[youngsuh]

@gcusello Thanks for taking the time to reply.  I'll give that a shot and get back to you.

[gcusello]

Hi @youngsuh,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

[youngsuh]

This is a note for people who visit this post.  Turn out that you have to send to the null queue then process the filter from the null queue.  But, the documentation or I was reading a differently Route and filter data.

As with other index-time field extractions, processing of transforms happens in the order that you specify them, from left to right. The key difference is the order in which you specify the stanzas. In this example, the setnull stanza must appear first in the list. This is because if you set it last, it matches all events and sends them to the nullQueue, and as it is the last transform, it effectively throws all of the events away, even those that previously matched the setparsing stanza.

the left to right is saying send everything to null queue then process the filter.  Hope that better for those are trying route data differently.  Oh, you don't have to write regex for the whole line of pattern instead write a unique portion to enforce like host or domain name in my case. 

[VatsalJagani]

@youngsuh - Your regex101 link does not seem to be valid anymore.

Also, just be sure does your log file name is just message or it message.log, or messages.log or message-2022-09-16.log???

If that's the case then you must specify the wildcard in the props.conf stanza.