Multiline Multivalue parsing

[aarcro]

I need to parse logs (windows events) that look roughly like this:

field1=[value1]
field2=[value2]
field3=[value3]
Description:
blah [subvalue 1.1] blah blah [subvalue 2.1] [subvalue 3.1]
blah [subvalue 1.2] blah blah [subvalue 2.2] [subvalue 3.2]
blah [subvalue 1.3] blah blah [subvalue 2.3] [subvalue 3.3]

I've looked at mvexpand and the docs about fields.conf, but I don't see how to get the values from different parts of the record. The above should be treated as 3 events which all have the same value for field[1-3] and different values for subvalue[1-3]

I already have regex's that will parse each line in the description portion.

[itinney · Accepted solution]

Yes it seems that the 's' modifier is on by default which has the effect that '.' will match new-lines. You can selectively turn off any of the 4 modifiers by prefixing them with a '-'.

Just to be clear, the 'm' modifier changes the behaviour of the anchors '^' and '$'. When 'm' is off, '^' means match the beginning of a string and '$' means match the end of a string. If the string happens to be a multi-line event then '^' matches the start of the event and '$' matches the end of the event (not the end of a line).

With the 'm' modifier turned on '^' matches the start of a line and '$' matches the end of a line. So '(?m)^foo.*bar$' will match the word 'foo' at the beginning of any line and then zero of more characters up to the string 'bar' at the end of the line (just before a newline character).

If you want '.' to match across multiple lines you use the 's' modifier. So '(?sm)^foo.*bar' will match the word 'foo' at the beginning of any line and then zero or more characters (including newline characters) greedily until it finds the word bar at the end of a line.

Spunk appear to have switched on the 's' modifier in 4.3? so that '.*' is always matching newlines. To turn this behaviour off and restore the default for PCRE you need to use the '-s' notation. So to turn on multi-line matching and turn off single-line mode you would say '^(?m-s)foo.*bar'

[aarcro]

Thank you so much! (?m-s) does get this working.

[aarcro]

Here's what I've got in my transforms.conf right now, but it's not working:

[section_name]
MV_ADD = true
SOURCE_KEY = _raw
REGEX =(^\w+\([^ ]+) (\w+) (.+) (((\d+) bytes) )?using (.+)$)+
FORMAT = user::$2 action::$3 path::$4 size::$6 method::$7

I've tried with and without (?m), but . still seems to match newlines. My records have between 1 and lots of lines that match REGEX, I want each match to add values to the multivalue fields: user, action, path, size and method.