topic Re: REGEX and NullQueue problem in Splunk Search

REGEX and NullQueue problem

echojacques — Mon, 28 Oct 2013 21:22:56 GMT

Hello,

I have setup a nullqueue to drop certain types of traffic. The traffic I want to drop is dest_port=53, any traffic from 10.10.10.10, and any traffic from the private IP address range: 169.254*. This is how I have my nullqueue setup:

In my /splunk/etc/system/local/transforms.conf I have:

[setnullitems]
REGEX=dest_port=53|10.10.10.10|169.254*
DEST_KEY=queue
FORMAT=nullQueue

In my /splunk/etc/system/local/props.conf I have:

TRANSFORMS-null=setnullitems

But the above doesn't work... does anyone know what I have done wrong? Thanks!

Re: REGEX and NullQueue problem

lguinn2 — Mon, 28 Oct 2013 21:39:24 GMT

I think you need a different regular expression. I don't think that one is doing what you want.

To test it, run this search (assuming that the events are ending up in the main index instead of being filtered):

index=main 
| regex _raw="dest_port=53|10.10.10.10|169.254*"

And I think you will find that it does not match any events. Try this instead, in your transforms.conf stanza:

REGEX=(?:dest_port\=53)|(?:10\.10\.10\.10)|(?:169\.254.*)

Again, to test this, run the following search

index=main 
| regex _raw="(?:dest_port\=53)|(?:10\.10\.10\.10)|(?:169\.254.*)"

Re: REGEX and NullQueue problem

echojacques — Mon, 28 Oct 2013 21:42:58 GMT

Thanks, I'll run the tests now and then report back soon!

Re: REGEX and NullQueue problem

echojacques — Mon, 28 Oct 2013 21:51:26 GMT

hmm they are both returning hundreds of matched events in the last 15 minutes...

Re: REGEX and NullQueue problem

lguinn2 — Mon, 28 Oct 2013 22:10:29 GMT

Are they the right events? The regular expression is looking for events that match one or more of the following patterns:

dest_port=53

10.10.10.10

169.254

These patterns can appear anywhere in the event and it should match. I would also search for these individually, just to see if they make sense - in other words, I would run three searches, one for each phrase.

Re: REGEX and NullQueue problem

echojacques — Mon, 28 Oct 2013 22:32:54 GMT

Thanks and yes, they are the right events. I think the problem is not the REGEX syntax but instead the nullqueue (which I've struggled with many times before)... is there anything special needed to include an OR "|" in the nullqueue? Or would it be better to just have multiple nullqueues instead of just putting all of the statements into one nullqueue?

Re: REGEX and NullQueue problem

echojacques — Mon, 28 Oct 2013 22:58:00 GMT

In your REGEX, what does the "?:" do?

Re: REGEX and NullQueue problem

lguinn2 — Tue, 29 Oct 2013 00:07:38 GMT

The ?: is part of the (?: ) pattern - it is called a "non-capturing group"

By making the sub-expressions into non-capturing groups, I am making it clear that the alternatives are between the sub-expressions. If you just use parentheses, that is a "capture group" in regular expressions, and that often means "field extraction" to Splunk - which is not what I want.

The | should work in the regular expression - you shouldn't need to do anything special.

Re: REGEX and NullQueue problem

lguinn2 — Tue, 29 Oct 2013 00:11:01 GMT

Re: REGEX and NullQueue problem

echojacques — Tue, 29 Oct 2013 14:44:44 GMT

Hi, thanks for the info and for teaching me about the (?:) non-capturing group syntax. I'm going to try a few things and then post my results here a little later today. Also, there are no other transforms with the classname null in my props.conf.

Re: REGEX and NullQueue problem

echojacques — Tue, 29 Oct 2013 21:19:34 GMT

So I'm not sure why, but after lots of testing, this is what ended up working for me:

REGEX=dest_port=53\b|10.10.10.10\b|169.254.*

I added the "\b" to specify port 53 and not 533, 5333, etc. Also, whenever I added the \escape characters in the IP address it would not work. Not sure why but at least it's working now. Thanks for your help!

Re: REGEX and NullQueue problem

lguinn2 — Tue, 29 Oct 2013 23:00:43 GMT

I just realized what part of the problem might have been. Sometimes Splunk gets confused when a regular expression contains an equals sign, since the equals sign is also part of the transforms.conf syntax. So watch out for that in the future!

Good idea with the \b

BTW, it shouldn't be a problem here, but remember that the period (.) is the wild-card character in regular expressions. For example 10z10b10c10 would actually match the regular expression string 10.10.10.10