topic Re: /var/log/messages filter null queue not filtering in Getting Data In

Why are /var/log/messages filter null queue not filtering?

youngsuh — Fri, 16 Sep 2022 14:15:26 GMT

Need regex & Null queue help to send events in /var/log/messages.

Here is regex101:

(IP & hostname randomized)

props.conf
[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing

transforms.conf
[setnull]
REGEX = \w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\w+\n
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\w{5}\d{4}\S\i.ab2.jone.com\s.+\n
DEST_KEY = queue
FORMAT = indexQueue

the regex not sending unwanted event in /var/log/message . I am doing the on HF before UF.

Re: /var/log/messages filter null queue not filtering

VatsalJagani — Fri, 16 Sep 2022 05:09:56 GMT

@youngsuh - Your regex101 link does not seem to be valid anymore.

Also, just be sure does your log file name is just message or it message.log, or messages.log or message-2022-09-16.log???

If that's the case then you must specify the wildcard in the props.conf stanza.

Re: /var/log/messages filter null queue not filtering

gcusello — Fri, 16 Sep 2022 05:47:56 GMT

Hi @youngsuh,

if you want to take from /var/log/messages only events containing "i.ab2.jone.com", you should try something like this:

props.conf [source::/var/log/messages] TRANSFORMS-set = setnull,setparsing transforms.conf [setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue [setparsing] REGEX = i\.ab2\.jone\.com DEST_KEY = queue FORMAT = indexQueue

Ciao.

Giuseppe

Re: /var/log/messages filter null queue not filtering

youngsuh — Fri, 16 Sep 2022 22:29:18 GMT

@gcusello Thanks for taking the time to reply. I'll give that a shot and get back to you.

Re: /var/log/messages filter null queue not filtering

gcusello — Sat, 17 Sep 2022 04:30:49 GMT

Hi @youngsuh,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

Re: /var/log/messages filter null queue not filtering

youngsuh — Mon, 19 Sep 2022 13:30:51 GMT

This is a note for people who visit this post. Turn out that you have to send to the null queue then process the filter from the null queue. But, the documentation or I was reading a differently Route and filter data.

As with other index-time field extractions, processing of transforms happens in the order that you specify them, from left to right. The key difference is the order in which you specify the stanzas. In this example, the setnull stanza must appear first in the list. This is because if you set it last, it matches all events and sends them to the nullQueue, and as it is the last transform, it effectively throws all of the events away, even those that previously matched the setparsing stanza.

the left to right is saying send everything to null queue then process the filter. Hope that better for those are trying route data differently. Oh, you don't have to write regex for the whole line of pattern instead write a unique portion to enforce like host or domain name in my case.