Exclude some events from being indexed

[pil321]

Calling all regex gurus!

I’m trying to drop all traffic with a certain IP (192.168.1.1) or a certain port number (123). This is what the log looks like:

2017-08-03 10:39:19,2017-08-03 10:39:19,0.000,192.168.1.1,192.168.6.225,123,123,,....

I found an answer for a way to do this (https://answers.splunk.com/answers/96/how-do-i-exclude-some-events-from-being-indexed-by-splunk.html).

This is what I have for my props.conf:

[source::/some/directory/in/splunk]
TRANSFORMS-set = null_1

This is transforms.conf:

[null_1]
REGEX = 192\.168.\1\.1
DEST_KEY = queue
FORMAT = nullQueue

I’m pretty sure the problem is with the regex, but I don’t have any regex skill whatsoever. Dropping all port 123 traffic would work as well.

[SteveSmi]

the issue is definitely your regex.

This part is wrong:

192\.168.\1\.1

\1 doesn’t make sense there unless you previously captured something. You just want to match the literal IP. To drop all events containing 192.168.1.1, your transforms.conf should look like this:

[null_1] REGEX = 192\.168\.1\.1 DEST_KEY = queue FORMAT = nullQueue

That will drop any event that contains that IP anywhere in the line.

If instead you want to drop all traffic with port 123, and assuming the port appears as a full field like in your example:

...,192.168.1.1,192.168.6.225,123,123,...

You can match it like this:

REGEX = ,123,

That makes sure you’re matching the port field and not something random like part of another number.

If you want to drop events that contain either that IP or port 123, you can combine them:

REGEX = 192\.168\.1\.1|,123,

That’s all you need.

[gcusello]

Hi pil321,
I usually use sourcetype in filters to be sure that it runs!
So try something like this:

in props.conf

[your_sourcetype]
TRANSFORMS-set-exclude=set_exclude,set_nullqueue

in transforms.conf

[set_exclude]
REGEX=.
DEST_KEY = queue
FORMAT = indexQueue
[set_nullqueue]
REGEX=(192\.168\.1\.1)|(,123,)
DEST_KEY=queue
FORMAT=nullQueue

Bye.
Giuseppe

[gcusello]

Hi pil321,
I usually use sourcetype in filters to be sure that it runs!
So try something like this:

in props.conf

[your_sourcetype]
TRANSFORMS-set-exclude=set_exclude,set_nullqueue

in transforms.conf

[set_exclude]
REGEX=.
DEST_KEY = queue
FORMAT = indexQueue
[set_nullqueue]
REGEX=(192\.168\.1\.1)|(,123,)
DEST_KEY=queue
FORMAT=nullQueue

Bye.
Giuseppe

[bheemireddi]

pil321,
anything with .* may be matching with lot more stuff than you think, to be precise if you just want to match with an IP address field, I wouldn't use .*
I just did a quick test and below regex should solve yours, if all you are looking to drop the events with that IP match 192\.168\.1\.1

[null_1]
REGEX = ,192\.168\.1\.1,
DEST_KEY = queue
FORMAT = nullQueue

[sbbadri · Accepted solution]

try this

[source::/some/directory/in/splunk]
TRANSFORMS-null = setnull

[setnull]
REGEX = \,192\.168\.1\.1\,

DEST_KEY = queue

FORMAT = nullQueue

---- or -----

[setnull]
REGEX =\d+-\d+-d+\s\d+:\d+:\d+\,\d+-\d+-d+\s\d+:\d+:\d+\, \,192\.168\.1\.1\,

DEST_KEY = queue

FORMAT = nullQueue

[pil321]

Sorry folks....the typo was on the code in the post...not on the actual configs!

This is what I have in the configs: 192\.168\.1\.1

[alemarzu]

This was configured on the indexer ?

[bheemireddi]

I didn't test it myself, but a quick spot you missed a "\" in before one of the dots. may be you can try this one. I added "," as well to make sure it is getting from the right place.

[null_1]
REGEX = ,192\.168\.1\.1,
DEST_KEY = queue
FORMAT = nullQueue

[FritzWittwer_ol]

your regexp has a typo, it should be:

REGEX = 192\\.168\\.1\\.1

regex010 ist one of the helpful online regular expressions checkers

[pil321]

Yep...I went there. The thing is...my expression also works on that site. The \. is meant to literally match the .

In your case the . is matching everything after the numbers....so your expression works as well.

I can give your expression a try and see.

[FritzWittwer_ol]

I'd try

REGEX = .\*192\\.168\\.1\\.1.\*

but the .* should not be needed, so eventually a config wich is not seen or overriden, did you try btool to verify the configuration?