Getting Data In

Why are /var/log/messages filter null queue not filtering?

youngsuh
Contributor

Need regex & Null queue help to send events in /var/log/messages.

Here is regex101:

regex101: build, test, and debug regex  

 (IP & hostname randomized)

props.conf
[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing

transforms.conf
[setnull]
REGEX = \w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\w+\n
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\w{5}\d{4}\S\i.ab2.jone.com\s.+\n
DEST_KEY = queue
FORMAT = indexQueue

the regex not sending unwanted event in /var/log/message .  I am doing the on HF before UF.

 

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @youngsuh,

if you want to take from /var/log/messages only events containing "i.ab2.jone.com", you should try something like this:

props.conf
[source::/var/log/messages]
TRANSFORMS-set = setnull,setparsing

transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = i\.ab2\.jone\.com
DEST_KEY = queue
FORMAT = indexQueue

Ciao.

Giuseppe

 

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @youngsuh,

if you want to take from /var/log/messages only events containing "i.ab2.jone.com", you should try something like this:

props.conf
[source::/var/log/messages]
TRANSFORMS-set = setnull,setparsing

transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = i\.ab2\.jone\.com
DEST_KEY = queue
FORMAT = indexQueue

Ciao.

Giuseppe

 

0 Karma

youngsuh
Contributor

@gcusello Thanks for taking the time to reply.  I'll give that a shot and get back to you.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @youngsuh,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

youngsuh
Contributor

This is a note for people who visit this post.  Turn out that you have to send to the null queue then process the filter from the null queue.  But, the documentation or I was reading a differently Route and filter data.

 

 

As with other index-time field extractions, processing of transforms happens in the order that you specify them, from left to right. The key difference is the order in which you specify the stanzas. In this example, the setnull stanza must appear first in the list. This is because if you set it last, it matches all events and sends them to the nullQueue, and as it is the last transform, it effectively throws all of the events away, even those that previously matched the setparsing stanza.

 

 

the left to right is saying send everything to null queue then process the filter.  Hope that better for those are trying route data differently.  Oh, you don't have to write regex for the whole line of pattern instead write a unique portion to enforce like host or domain name in my case. 

 

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

@youngsuh - Your regex101 link does not seem to be valid anymore.

Also, just be sure does your log file name is just message or it message.log, or messages.log or message-2022-09-16.log???

If that's the case then you must specify the wildcard in the props.conf stanza.

 

Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...