Splunk Search

How to extract message fields from logs with rex?

Explorer

Hi all,

I want to extract data from a log which is like that :

2014-21-08 07:10:57,603.812  - DEBUG- (pid: 12727 tid: 13196) blablabla_message_1
2014-21-08 07:10:27,118.983  - DEBUG- blablabla_message_2

I dont want the "pid" and "tid" value but i would like that blablabla_message_1 and blablabla_message_2 are in the same field.

If i do this :

| rex ".+- (?<GCM_type_name>[A-Z]+)\s*- \(pid\: [0-9]+ tid\: [0-9]+\)(?<GCM_message>.+)"

I get blablabla_message_1 but not blablabla_message_2

And if i do just this :

| rex ".+- (?<type_name>[A-Z]+)\s*- (?<message>.+)"

I get blablabla_message_2 but also "(pid: 12727 tid: 13196) blablabla_message_1" 😞

Is it possible ?

1 Solution

SplunkTrust
SplunkTrust

You could do something like this:

... | rex "\s-\s+(?<log_level>[A-Z]+)\s*-\s+(?:\([^)]*\))?\s*(?<message>.+)"

That'll cut out the first parenthesized group at the beginning if there is one, and proceeds as normal if there isn't one.
This assumes that there is no parentheses within that parenthesized group, and that the message itself cannot start with a parenthesized group that's part of the message.

If one of those assumptions doesn't hold then I'm sure the regex could be made more complicated to allow for your reality.

View solution in original post

SplunkTrust
SplunkTrust

You could do something like this:

... | rex "\s-\s+(?<log_level>[A-Z]+)\s*-\s+(?:\([^)]*\))?\s*(?<message>.+)"

That'll cut out the first parenthesized group at the beginning if there is one, and proceeds as normal if there isn't one.
This assumes that there is no parentheses within that parenthesized group, and that the message itself cannot start with a parenthesized group that's part of the message.

If one of those assumptions doesn't hold then I'm sure the regex could be made more complicated to allow for your reality.

View solution in original post

SplunkTrust
SplunkTrust

Sure.

\( looks for a literal opening parenthesis
[^)]* looks for zero or more characters that aren't a closing parenthesis
\) looks for a literal closing parenthesis
(?:...) wraps all of the above in a noncapturing group
? at the end makes that noncapturing group optional, appearing zero or one time

SplunkTrust
SplunkTrust

For interactive explanations you can use http://regexr.com/ - just make sure not to use named capturing groups, those aren't supported there.

Explorer

Ohhhh, ok ! Nice 🙂

Really thank you, it's working and I understand better regex now 🙂

Explorer

Ok, thank you, I will try it.

But can you explain me :

(?:\([^)]*\))?

My incomprehension is probably for the "?" or the "^" in this sequence

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!