topic Help with REGEX for data filter in Splunk Search

Help with REGEX for data filter

nce054 — Fri, 19 Jun 2015 14:24:42 GMT

In my transforms.conf I currently have

[filter-marimba]
REGEX=^(?!\[[^\]]+\]\s+-\s+warning.*)
DEST_KEY = queue
FORMAT = nullQueue

It should be catching any message that doesn't have 'warning' in it.
This is data it should be allowing

[19/Jun/2015:09:07:36 -0500] - warning nce054_a 50012 Common Reboot Service is disabled.

This is some data it should be sending to the null queue

[19/Jun/2015:09:07:42 -0500] - info nce054_a 9236 Adapter stopped, packaged channel URL: http://mrbamtx:5282/Root/Prod/DeskMgmt/PrintQMigration

Any advice?

Re: Help with REGEX for data filter

woodcock — Fri, 19 Jun 2015 14:30:44 GMT

Is this still true?

inputs.conf

[monitor://C:\Windows\.marimba\MarimbaEndpointTuner\history-y*.log]
sourcetype = marimba

props.conf

[marimba]
TRANSFORMS-mfilter=filter-marimba

Re: Help with REGEX for data filter

nce054 — Fri, 19 Jun 2015 14:34:40 GMT

Yes, both of those are the same. Sorry for starting a new thread, I felt like it was getting congested. Trying the REGEX = ^# right now.

Re: Help with REGEX for data filter

nce054 — Fri, 19 Jun 2015 14:38:04 GMT

This REGEX=^# is essentially saying that any input string that matches this (anything that starts with #) should be sent to the nullQueue, i.e. not sent to Indexer. Is that correct? When I put the expression ^# into https://regex101.com/ and insert one of my pieces of data, like `#[19/Jun/2015:09:31:29 -0500]

ret = 0` , it says that there is a match, but no groups were extracted.

Re: Help with REGEX for data filter

woodcock — Fri, 19 Jun 2015 14:41:37 GMT

ARGH! Then your example data (admittedly from previous question) was not written correctly by you!
Try this one:

REGEX= \]\s+#

Re: Help with REGEX for data filter

nce054 — Fri, 19 Jun 2015 14:51:30 GMT

My apologies, I messed up on that one. The basic point of this is trying to remove events where the only content is comments. Sorry for the confusion. According to https://regex101.com , that pattern wouldn't match the junk data. If I'm wrong, please let me know. My experience with REGEX is extremely limited. Thanks for your time.

Re: Help with REGEX for data filter

nce054 — Fri, 19 Jun 2015 14:55:18 GMT

I pushed this REGEX out in the transforms.conf at 9:46, and I still found this at 9:52 on the Search Head :
#[19/Jun/2015:09:52:42 -0500]
#ret = 0

Re: Help with REGEX for data filter

woodcock — Fri, 19 Jun 2015 14:55:24 GMT

In this use-case, the manner of RegEx is not anchored but your tool must be. To test with your tool, add .* to the front and back ends of the RegEx, like this: .*\]\s+#.*

Re: Help with REGEX for data filter

woodcock — Fri, 19 Jun 2015 15:06:03 GMT

Is there really a leading '#' character? You really have to make up your mind about what your data looks like! In any case, I am worn out. I have used this basic configuration a dozen time and it works great so I am convinced the problem is the RegEx but I am worn out and will let somebody else comment.

Re: Help with REGEX for data filter

nce054 — Fri, 19 Jun 2015 15:08:55 GMT

Yes, there is really a leading '#' character. Thanks for the effort.

Re: Help with REGEX for data filter

ccraig42 — Fri, 19 Jun 2015 21:47:28 GMT

Are you saying that regular expression is sending the one with "warning" to the null queue or not sending the other one? I would have written a match to send '^[[^]]+]\s+-\s+warning' to a real queue instead of using a negative lookahead assertion, and then everything else to the nullQueue, but the regex you have should work on those example lines.

Re: Help with REGEX for data filter

woodcock — Fri, 19 Jun 2015 22:11:33 GMT

It takes 2 steps to do it that way which is why I showed you a 1-step design that throws away what matches.

Re: Help with REGEX for data filter

ccraig42 — Fri, 19 Jun 2015 22:18:33 GMT

Sorry, I'm not trying to say it's a bad regex, that's exactly my problem. It's not what I would have used, but it should work with the example data he has, which makes it rather difficult to produce a regular expression that works on the data that's failing, but isn't provided.

Re: Help with REGEX for data filter

rkent — Tue, 23 Jun 2015 00:25:09 GMT

This regex uses a negative lookahead to match all lines (in the same format as the samples you've provided) to not match any event that has either the words warning or info in them:

^(?:[[^]]+]\s+-)\s+(?!warning).$
^(?:[[^]]+]\s+-)\s+(?!info).$

You can test the first of the regexes at the following link: http://regexr.com/3b8m7

To test, remove one of the letters from the word "warning" and you will see it instantly match the sample event.

Re: Help with REGEX for data filter

woodcock — Sun, 28 Jun 2015 21:19:05 GMT

I just tested this configuration and it DOES work:

[filter-marimba]
REGEX=^\[[^\]]+\]\s+-\s+(?!warning).*
DEST_KEY = queue
FORMAT = nullQueue

It should be trashing any message that doesn't have 'warning' in that particular spot.