REGEX and NullQueue problem

[echojacques]

Hello,

I have setup a nullqueue to drop certain types of traffic. The traffic I want to drop is dest_port=53, any traffic from 10.10.10.10, and any traffic from the private IP address range: 169.254*. This is how I have my nullqueue setup:

In my /splunk/etc/system/local/transforms.conf I have:

[setnullitems]
REGEX=dest_port=53|10.10.10.10|169.254*
DEST_KEY=queue
FORMAT=nullQueue

In my /splunk/etc/system/local/props.conf I have:

TRANSFORMS-null=setnullitems

But the above doesn't work... does anyone know what I have done wrong? Thanks!

[lguinn2 · Accepted solution]

I think you need a different regular expression. I don't think that one is doing what you want.

To test it, run this search (assuming that the events are ending up in the main index instead of being filtered):

index=main 
| regex _raw="dest_port=53|10.10.10.10|169.254*"

And I think you will find that it does not match any events. Try this instead, in your transforms.conf stanza:

REGEX=(?:dest_port\=53)|(?:10\.10\.10\.10)|(?:169\.254.*)

Again, to test this, run the following search

index=main 
| regex _raw="(?:dest_port\=53)|(?:10\.10\.10\.10)|(?:169\.254.*)"

[lguinn2]

I just realized what part of the problem might have been. Sometimes Splunk gets confused when a regular expression contains an equals sign, since the equals sign is also part of the transforms.conf syntax. So watch out for that in the future!

Good idea with the \b

BTW, it shouldn't be a problem here, but remember that the period (.) is the wild-card character in regular expressions. For example 10z10b10c10 would actually match the regular expression string 10.10.10.10

[echojacques]

So I'm not sure why, but after lots of testing, this is what ended up working for me:

REGEX=dest_port=53\b|10.10.10.10\b|169.254.*

I added the "\b" to specify port 53 and not 533, 5333, etc. Also, whenever I added the \escape characters in the IP address it would not work. Not sure why but at least it's working now. Thanks for your help!

[echojacques]

Hi, thanks for the info and for teaching me about the (?:) non-capturing group syntax. I'm going to try a few things and then post my results here a little later today. Also, there are no other transforms with the classname null in my props.conf.

[lguinn2]

Other questions:

Can you show the complete stanza in props.conf that contains

TRANSFORMS-null=setnullitems

Are there any other transforms with the classname null? This name needs to be unique.

[lguinn2]

The ?: is part of the (?: ) pattern - it is called a "non-capturing group"

By making the sub-expressions into non-capturing groups, I am making it clear that the alternatives are between the sub-expressions. If you just use parentheses, that is a "capture group" in regular expressions, and that often means "field extraction" to Splunk - which is not what I want.

The | should work in the regular expression - you shouldn't need to do anything special.

[echojacques]

In your REGEX, what does the "?:" do?

[echojacques]

Thanks and yes, they are the right events. I think the problem is not the REGEX syntax but instead the nullqueue (which I've struggled with many times before)... is there anything special needed to include an OR "|" in the nullqueue? Or would it be better to just have multiple nullqueues instead of just putting all of the statements into one nullqueue?

[lguinn2]

Are they the right events? The regular expression is looking for events that match one or more of the following patterns:

dest_port=53

10.10.10.10

169.254

These patterns can appear anywhere in the event and it should match. I would also search for these individually, just to see if they make sense - in other words, I would run three searches, one for each phrase.

[echojacques]

hmm they are both returning hundreds of matched events in the last 15 minutes...

[echojacques]

Thanks, I'll run the tests now and then report back soon!