Getting Data In

Why are /var/log/messages filter null queue not filtering?

youngsuh
Contributor

Need regex & Null queue help to send events in /var/log/messages.

Here is regex101:

regex101: build, test, and debug regex  

 (IP & hostname randomized)

props.conf
[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing

transforms.conf
[setnull]
REGEX = \w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\w+\n
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\s\w{5}\d{4}\S\i.ab2.jone.com\s.+\n
DEST_KEY = queue
FORMAT = indexQueue

the regex not sending unwanted event in /var/log/message .  I am doing the on HF before UF.

 

0 Karma
1 Solution

gcusello
Esteemed Legend

Hi @youngsuh,

if you want to take from /var/log/messages only events containing "i.ab2.jone.com", you should try something like this:

props.conf
[source::/var/log/messages]
TRANSFORMS-set = setnull,setparsing

transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = i\.ab2\.jone\.com
DEST_KEY = queue
FORMAT = indexQueue

Ciao.

Giuseppe

 

View solution in original post

0 Karma

gcusello
Esteemed Legend

Hi @youngsuh,

if you want to take from /var/log/messages only events containing "i.ab2.jone.com", you should try something like this:

props.conf
[source::/var/log/messages]
TRANSFORMS-set = setnull,setparsing

transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = i\.ab2\.jone\.com
DEST_KEY = queue
FORMAT = indexQueue

Ciao.

Giuseppe

 

0 Karma

youngsuh
Contributor

@gcusello Thanks for taking the time to reply.  I'll give that a shot and get back to you.

0 Karma

gcusello
Esteemed Legend

Hi @youngsuh,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

youngsuh
Contributor

This is a note for people who visit this post.  Turn out that you have to send to the null queue then process the filter from the null queue.  But, the documentation or I was reading a differently Route and filter data.

 

 

As with other index-time field extractions, processing of transforms happens in the order that you specify them, from left to right. The key difference is the order in which you specify the stanzas. In this example, the setnull stanza must appear first in the list. This is because if you set it last, it matches all events and sends them to the nullQueue, and as it is the last transform, it effectively throws all of the events away, even those that previously matched the setparsing stanza.

 

 

the left to right is saying send everything to null queue then process the filter.  Hope that better for those are trying route data differently.  Oh, you don't have to write regex for the whole line of pattern instead write a unique portion to enforce like host or domain name in my case. 

 

0 Karma

VatsalJagani
Super Champion

@youngsuh - Your regex101 link does not seem to be valid anymore.

Also, just be sure does your log file name is just message or it message.log, or messages.log or message-2022-09-16.log???

If that's the case then you must specify the wildcard in the props.conf stanza.

 

youngsuh
Contributor

the file is just message without the extension.

https://regex101.com/r/HrrqiV/1

Here is the regex101 update.  Apparently, I delete. 

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...