Getting Data In

Can I view related parts of a multiline log entry?

ripper234
Explorer

Some of the long entries my app makes are composed of multiple lines.
I would like to keep it this way (a log line can be 10+ lines sometimes, has lots of info that can't be condensed into a single line).

Is there a way, given a specific text match, to view text lines near it?

E.g. supposed that my log format looks like this:

2013-01-07 13:28:27,325 INFO  (LoggerName) Something is wrong with Website http://foo.com/
Now follow a few important details.
A few more details.
2013-01-07 13:28:27,325 INFO  (LoggerName) Another log entry

If I search in splunk for "foo.com", and only find the first line without the details - how can I view the rest of the details?

I prefer a solution that won't force me to change the format of my log messages.

Tags (1)
0 Karma
1 Solution

emiller42
Motivator

Are the multi-line events being broken into multiple events in Splunk?

Event 1:
2013-01-07 13:28:27,325 INFO  (LoggerName) Something is wrong with Website http://foo.com/
Event 2:
Now follow a few important details.
Event 3:
A few more details.
Event 4:
2013-01-07 13:28:27,325 INFO  (LoggerName) Another log entry

If so, you can fix that where it keeps the whole thing as one event.

In your props.conf, you want to add the following config to the appropriate stanza for this sourcetype:

LINE_BREAKER = ([\r\n]+)\d+-\d+-\d+\s\d+:\d+:\d+,\d+ 
SHOULD_LINEMERGE = false

This tells it not to automatically break events, and instead only break when it encounters a new line starting with a timestamp. (What the regex matches)

Then when new log lines are indexed, multi-line events will be kept as a single event, keeping the context you want:

Event 1:
2013-01-07 13:28:27,325 INFO  (LoggerName) Something is wrong with Website http://foo.com/
Now follow a few important details.
A few more details.
Event 2:
2013-01-07 13:28:27,325 INFO  (LoggerName) Another log entry

View solution in original post

emiller42
Motivator

Are the multi-line events being broken into multiple events in Splunk?

Event 1:
2013-01-07 13:28:27,325 INFO  (LoggerName) Something is wrong with Website http://foo.com/
Event 2:
Now follow a few important details.
Event 3:
A few more details.
Event 4:
2013-01-07 13:28:27,325 INFO  (LoggerName) Another log entry

If so, you can fix that where it keeps the whole thing as one event.

In your props.conf, you want to add the following config to the appropriate stanza for this sourcetype:

LINE_BREAKER = ([\r\n]+)\d+-\d+-\d+\s\d+:\d+:\d+,\d+ 
SHOULD_LINEMERGE = false

This tells it not to automatically break events, and instead only break when it encounters a new line starting with a timestamp. (What the regex matches)

Then when new log lines are indexed, multi-line events will be kept as a single event, keeping the context you want:

Event 1:
2013-01-07 13:28:27,325 INFO  (LoggerName) Something is wrong with Website http://foo.com/
Now follow a few important details.
A few more details.
Event 2:
2013-01-07 13:28:27,325 INFO  (LoggerName) Another log entry

ripper234
Explorer

Interesting direction, I think it will do the trick. I think the lines are all separate events.

Will try it out when I get the chance - thanks for the quick answer!

0 Karma
Get Updates on the Splunk Community!

Building Reliable Asset and Identity Frameworks in Splunk ES

 Accurate asset and identity resolution is the backbone of security operations. Without it, alerts are ...

Cloud Monitoring Console - Unlocking Greater Visibility in SVC Usage Reporting

For Splunk Cloud customers, understanding and optimizing Splunk Virtual Compute (SVC) usage and resource ...

Automatic Discovery Part 3: Practical Use Cases

If you’ve enabled Automatic Discovery in your install of the Splunk Distribution of the OpenTelemetry ...