Getting Data In

Why are /var/log/messages filter null queue not filtering?

youngsuh
Contributor

Need regex & Null queue help to send events in /var/log/messages.

Here is regex101:

regex101: build, test, and debug regex  

 (IP & hostname randomized)

props.conf
[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing

transforms.conf
[setnull]
REGEX = \w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\w+\n
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\w{5}\d{4}\S\i.ab2.jone.com\s.+\n
DEST_KEY = queue
FORMAT = indexQueue

the regex not sending unwanted event in /var/log/message .  I am doing the on HF before UF.

 

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @youngsuh,

if you want to take from /var/log/messages only events containing "i.ab2.jone.com", you should try something like this:

props.conf
[source::/var/log/messages]
TRANSFORMS-set = setnull,setparsing

transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = i\.ab2\.jone\.com
DEST_KEY = queue
FORMAT = indexQueue

Ciao.

Giuseppe

 

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @youngsuh,

if you want to take from /var/log/messages only events containing "i.ab2.jone.com", you should try something like this:

props.conf
[source::/var/log/messages]
TRANSFORMS-set = setnull,setparsing

transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = i\.ab2\.jone\.com
DEST_KEY = queue
FORMAT = indexQueue

Ciao.

Giuseppe

 

0 Karma

youngsuh
Contributor

@gcusello Thanks for taking the time to reply.  I'll give that a shot and get back to you.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @youngsuh,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

youngsuh
Contributor

This is a note for people who visit this post.  Turn out that you have to send to the null queue then process the filter from the null queue.  But, the documentation or I was reading a differently Route and filter data.

 

 

As with other index-time field extractions, processing of transforms happens in the order that you specify them, from left to right. The key difference is the order in which you specify the stanzas. In this example, the setnull stanza must appear first in the list. This is because if you set it last, it matches all events and sends them to the nullQueue, and as it is the last transform, it effectively throws all of the events away, even those that previously matched the setparsing stanza.

 

 

the left to right is saying send everything to null queue then process the filter.  Hope that better for those are trying route data differently.  Oh, you don't have to write regex for the whole line of pattern instead write a unique portion to enforce like host or domain name in my case. 

 

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

@youngsuh - Your regex101 link does not seem to be valid anymore.

Also, just be sure does your log file name is just message or it message.log, or messages.log or message-2022-09-16.log???

If that's the case then you must specify the wildcard in the props.conf stanza.

 

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...