Getting Data In

Filtering unstructured multiline events with regular expression

msarro
Builder

This is hopefully a simple question but I couldn't find the answer in transforms.conf. I am receiving an EXTREMELY high volume data source (network protocols dump). Every event averages 30-60 lines. There are a few conditions where I would like to keep this data (the entire record), everything else should go to the nullQueue.

So, is there a way to specify to trash every event that doesn't match a particular regex/set of regex? The regexes could be found in different lines of the file depending on the network protocol.

Also, if I used a regex which specifies line start/linebreak, will this only look at the very first line of the event?

I have looked at http://docs.splunk.com/Documentation/Splunk/4.3.1/Deploy/Routeandfilterdatad#Keep_specific_events_an... but it doesn't seem to say how regular expressions work in multiline events. Will a regex save an entire event if the regex matches any part of any line in the event? Or does it only watch the first line?

Tags (1)
0 Karma

lguinn2
Legend

For multiline events, put (?m) at the beginning of your regular expression.This tells Splunk that it should look for matches across all the lines of the event.

Use that along with with the directions in the manual - it looks like you've found them.

Also, there are tons of references on regular expressions. Here is a personal favorite of mine:

http://www.regular-expressions.info

0 Karma

johandk
Path Finder

This should help:

http://docs.splunk.com/Documentation/Splunk/latest/Data/Indexmulti-lineevents

For multiline events your regex should maybe look something like this:

(m?)^my regex here\b

See here:

http://splunk-base.splunk.com/answers/38753/regex-for-multiline-events

http://splunk-base.splunk.com/answers/20/what-does-m-mean-at-the-beginning-of-a-regex

Also, when you are writing a regex to send events to the nullQueue Splunk will save the entire event if the regex does not match.

0 Karma

Jordan_Brough
Path Finder

The code example should have (?m) instead of (m?)

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.