Exclude some events from being indexed

[pil321]

Calling all regex gurus!

I’m trying to drop all traffic with a certain IP (192.168.1.1) or a certain port number (123). This is what the log looks like:

2017-08-03 10:39:19,2017-08-03 10:39:19,0.000,192.168.1.1,192.168.6.225,123,123,,....

I found an answer for a way to do this (https://answers.splunk.com/answers/96/how-do-i-exclude-some-events-from-being-indexed-by-splunk.html).

This is what I have for my props.conf:

[source::/some/directory/in/splunk]
TRANSFORMS-set = null_1

This is transforms.conf:

[null_1]
REGEX = 192\.168.\1\.1
DEST_KEY = queue
FORMAT = nullQueue

I’m pretty sure the problem is with the regex, but I don’t have any regex skill whatsoever. Dropping all port 123 traffic would work as well.

[gcusello]

Hi pil321,
I usually use sourcetype in filters to be sure that it runs!
So try something like this:

in props.conf

[your_sourcetype]
TRANSFORMS-set-exclude=set_exclude,set_nullqueue

in transforms.conf

[set_exclude]
REGEX=.
DEST_KEY = queue
FORMAT = indexQueue
[set_nullqueue]
REGEX=(192\.168\.1\.1)|(,123,)
DEST_KEY=queue
FORMAT=nullQueue

Bye.
Giuseppe

[gcusello]

Hi pil321,
I usually use sourcetype in filters to be sure that it runs!
So try something like this:

in props.conf

[your_sourcetype]
TRANSFORMS-set-exclude=set_exclude,set_nullqueue

in transforms.conf

[set_exclude]
REGEX=.
DEST_KEY = queue
FORMAT = indexQueue
[set_nullqueue]
REGEX=(192\.168\.1\.1)|(,123,)
DEST_KEY=queue
FORMAT=nullQueue

Bye.
Giuseppe

[bheemireddi]

pil321,
anything with .* may be matching with lot more stuff than you think, to be precise if you just want to match with an IP address field, I wouldn't use .*
I just did a quick test and below regex should solve yours, if all you are looking to drop the events with that IP match 192\.168\.1\.1

[null_1]
REGEX = ,192\.168\.1\.1,
DEST_KEY = queue
FORMAT = nullQueue

[sbbadri · Accepted solution]

try this

[source::/some/directory/in/splunk]
TRANSFORMS-null = setnull

[setnull]
REGEX = \,192\.168\.1\.1\,

DEST_KEY = queue

FORMAT = nullQueue

---- or -----

[setnull]
REGEX =\d+-\d+-d+\s\d+:\d+:\d+\,\d+-\d+-d+\s\d+:\d+:\d+\, \,192\.168\.1\.1\,

DEST_KEY = queue

FORMAT = nullQueue

[pil321]

Sorry folks....the typo was on the code in the post...not on the actual configs!

This is what I have in the configs: 192\.168\.1\.1

[alemarzu]

This was configured on the indexer ?

[bheemireddi]

I didn't test it myself, but a quick spot you missed a "\" in before one of the dots. may be you can try this one. I added "," as well to make sure it is getting from the right place.

[null_1]
REGEX = ,192\.168\.1\.1,
DEST_KEY = queue
FORMAT = nullQueue

[FritzWittwer_ol]

your regexp has a typo, it should be:

REGEX = 192\\.168\\.1\\.1

regex010 ist one of the helpful online regular expressions checkers

[pil321]

Yep...I went there. The thing is...my expression also works on that site. The \. is meant to literally match the .

In your case the . is matching everything after the numbers....so your expression works as well.

I can give your expression a try and see.

[FritzWittwer_ol]

I'd try

REGEX = .\*192\\.168\\.1\\.1.\*

but the .* should not be needed, so eventually a config wich is not seen or overriden, did you try btool to verify the configuration?