Splunk Search

How to split multi-line events at search time?

aa123s
Explorer

I have events that look like this:

[abc] logline1
[def] logline 2
[ghi] logline 3

and I would like to split those events at search time into 3 single line events.
Is that possible?

Thanks!

P.S.
I Know this should be done at Indexer / Heavy Forwarder level using LINE_BREAKER, but that's not an option at this time.

1 Solution

adonio
Ultra Champion

msquicc
Path Finder

here's what I came up with.  seems to work pretty well without modifying the data:

 

| makeresults | eval _raw = "[abc] logline1
[def] logline 2
[ghi] logline 3"


| eval raw=_raw
| makemv tokenizer="(.*(\r\n|\r|\n|$))" raw
| mvexpand raw
| rename raw as _raw
0 Karma

micahkemp
Champion

Referring to your previous question:

https://answers.splunk.com/answers/618398/why-is-splunk-not-breaking-each-log-line-into-sing.html#an...

I strongly suggest working now to get these logs indexed properly instead of trying to solve this problem at search time. You will end up being frustrated time and time again if your events are not indexed properly.

0 Karma

aa123s
Explorer

I got that, Thanks. We are already working to add correct indexing at forwarder level. In the mean time, however, we need this workaround.

0 Karma

adonio
Ultra Champion

hello there,

maybe try the mvexpand command
check i tout:
https://docs.splunk.com/Documentation/Splunk/7.0.2/SearchReference/Mvexpand

aa123s
Explorer

Before posting I tried this:
| rex mode=sed "s/([\r\n]+)/##LF##/g" | makemv _raw delim="##LF##" | mvexpand _raw

but I couldn't make it work. Events are joined in a long string separated by ##LF##, but then those lines don't split back into separate events

0 Karma

somesoni2
Revered Legend

Try like this. The mvexpand command doesn't seem to work with fields starting with underscore.

your base search | rex mode=sed "s/([\r\n]+)/##LF##/g" | makemv _raw delim="##LF##" | rename _raw as raw | mvexpand raw | rename raw as _raw

aa123s
Explorer

Super! Its almost working: the remaining problem is that lines are being re-grouped in reverse order... Could that be fixed? Thanks!

0 Karma

aa123s
Explorer

I'm sorry my comment was incomplete. I meant rows are being re-grouped in reverse order when I pipe the output of your solution to transaction ...
It normally doesn't happen

0 Karma

aa123s
Explorer

I ended up adding | reverse at the end... go figure why that happens!...
Thanks a lot!

0 Karma
Get Updates on the Splunk Community!

Splunk Enterprise Security 8.x: The Essential Upgrade for Threat Detection, ...

 Prepare to elevate your security operations with the powerful upgrade to Splunk Enterprise Security 8.x! This ...

Get Early Access to AI Playbook Authoring: Apply for the Alpha Private Preview ...

Passionate about security automation? Apply now to our AI Playbook Authoring Alpha private preview ...

Reduce and Transform Your Firewall Data with Splunk Data Management

Managing high-volume firewall data has always been a challenge. Noisy events and verbose traffic logs often ...