topic Re: Filtering unstructured multiline events with regular expression in Getting Data In

Filtering unstructured multiline events with regular expression

msarro — Fri, 24 Aug 2012 16:48:43 GMT

This is hopefully a simple question but I couldn't find the answer in transforms.conf. I am receiving an EXTREMELY high volume data source (network protocols dump). Every event averages 30-60 lines. There are a few conditions where I would like to keep this data (the entire record), everything else should go to the nullQueue.

So, is there a way to specify to trash every event that doesn't match a particular regex/set of regex? The regexes could be found in different lines of the file depending on the network protocol.

Also, if I used a regex which specifies line start/linebreak, will this only look at the very first line of the event?

I have looked at http://docs.splunk.com/Documentation/Splunk/4.3.1/Deploy/Routeandfilterdatad#Keep_specific_events_and_discard_the_rest but it doesn't seem to say how regular expressions work in multiline events. Will a regex save an entire event if the regex matches any part of any line in the event? Or does it only watch the first line?

Re: Filtering unstructured multiline events with regular expression

johandk — Fri, 24 Aug 2012 18:19:48 GMT

This should help:

http://docs.splunk.com/Documentation/Splunk/latest/Data/Indexmulti-lineevents

For multiline events your regex should maybe look something like this:

(m?)^my regex here\b

See here:

http://splunk-base.splunk.com/answers/38753/regex-for-multiline-events

http://splunk-base.splunk.com/answers/20/what-does-m-mean-at-the-beginning-of-a-regex

Also, when you are writing a regex to send events to the nullQueue Splunk will save the entire event if the regex does not match.

Re: Filtering unstructured multiline events with regular expression

lguinn2 — Fri, 24 Aug 2012 18:24:24 GMT

For multiline events, put (?m) at the beginning of your regular expression.This tells Splunk that it should look for matches across all the lines of the event.

Use that along with with the directions in the manual - it looks like you've found them.

Also, there are tons of references on regular expressions. Here is a personal favorite of mine:

http://www.regular-expressions.info

Re: Filtering unstructured multiline events with regular expression

Jordan_Brough — Fri, 28 Jun 2013 04:12:10 GMT

The code example should have (?m) instead of (m?)