Filtering unstructured multiline events with regular expression

[msarro]

This is hopefully a simple question but I couldn't find the answer in transforms.conf. I am receiving an EXTREMELY high volume data source (network protocols dump). Every event averages 30-60 lines. There are a few conditions where I would like to keep this data (the entire record), everything else should go to the nullQueue.

So, is there a way to specify to trash every event that doesn't match a particular regex/set of regex? The regexes could be found in different lines of the file depending on the network protocol.

Also, if I used a regex which specifies line start/linebreak, will this only look at the very first line of the event?

I have looked at http://docs.splunk.com/Documentation/Splunk/4.3.1/Deploy/Routeandfilterdatad#Keep_specific_events_an... but it doesn't seem to say how regular expressions work in multiline events. Will a regex save an entire event if the regex matches any part of any line in the event? Or does it only watch the first line?

[lguinn2]

For multiline events, put (?m) at the beginning of your regular expression.This tells Splunk that it should look for matches across all the lines of the event.

Use that along with with the directions in the manual - it looks like you've found them.

Also, there are tons of references on regular expressions. Here is a personal favorite of mine:

http://www.regular-expressions.info

[johandk]

This should help:

http://docs.splunk.com/Documentation/Splunk/latest/Data/Indexmulti-lineevents

For multiline events your regex should maybe look something like this:

(m?)^my regex here\b

See here:

http://splunk-base.splunk.com/answers/38753/regex-for-multiline-events

http://splunk-base.splunk.com/answers/20/what-does-m-mean-at-the-beginning-of-a-regex

Also, when you are writing a regex to send events to the nullQueue Splunk will save the entire event if the regex does not match.

[Jordan_Brough]

The code example should have (?m) instead of (m?)