How to extract message fields from logs with rex?

[splunksogetiht]

Hi all,

I want to extract data from a log which is like that :

2014-21-08 07:10:57,603.812  - DEBUG- (pid: 12727 tid: 13196) blablabla_message_1
2014-21-08 07:10:27,118.983  - DEBUG- blablabla_message_2

I dont want the "pid" and "tid" value but i would like that blablabla_message_1 and blablabla_message_2 are in the same field.

If i do this :

| rex ".+- (?<GCM_type_name>[A-Z]+)\s*- \(pid\: [0-9]+ tid\: [0-9]+\)(?<GCM_message>.+)"

I get blablabla_message_1 but not blablabla_message_2

And if i do just this :

| rex ".+- (?<type_name>[A-Z]+)\s*- (?<message>.+)"

I get blablabla_message_2 but also "(pid: 12727 tid: 13196) blablabla_message_1" 😞

Is it possible ?

[martin_mueller · Accepted solution]

You could do something like this:

... | rex "\s-\s+(?<log_level>[A-Z]+)\s*-\s+(?:\([^)]*\))?\s*(?<message>.+)"

That'll cut out the first parenthesized group at the beginning if there is one, and proceeds as normal if there isn't one.
This assumes that there is no parentheses within that parenthesized group, and that the message itself cannot start with a parenthesized group that's part of the message.

If one of those assumptions doesn't hold then I'm sure the regex could be made more complicated to allow for your reality.

[martin_mueller]

Sure.

\( looks for a literal opening parenthesis
[^)]* looks for zero or more characters that aren't a closing parenthesis
\) looks for a literal closing parenthesis
(?:...) wraps all of the above in a noncapturing group
? at the end makes that noncapturing group optional, appearing zero or one time

[martin_mueller]

For interactive explanations you can use http://regexr.com/ - just make sure not to use named capturing groups, those aren't supported there.

[splunksogetiht]

Ohhhh, ok ! Nice 🙂

Really thank you, it's working and I understand better regex now 🙂

[splunksogetiht]

Ok, thank you, I will try it.

But can you explain me :

(?:\([^)]*\))?

My incomprehension is probably for the "?" or the "^" in this sequence