Splunk Search

How do I simulate LINE_BREAKER at search time?

Jason
Motivator

I have some data that has been ingested quickly/badly, so there are multiple lines per event. Rather than reindex it, is there a way I can quickly break it up in a search, similar to LINE_BREAKER at parse time?

This is one event. I would like five:

20.30.40 this is an event -- The date is 6 January, 2017 --
20.30.41 this is another event
20.30.42 this is a multiline event?
  see?
  it's really a multiline event
  I promise.
20.30.43 this is a fourth event
20.30.44 and a final one
0 Karma
1 Solution

Jason
Motivator

Yes, you can use rex and multivalued fields to select new "raw" texts and expand them into new events. The fun is getting the regular expression right so you can use it like LINE_BREAKER. Here I am using a few things:

(?sm) -- s for "dot-all", so .*? matches newlines, and m for multiline, so ^ matches the beginning of a line instead of the beginning of an event
\d\d\.\d\d\.\d\d\s -- this would be the unique string, appearing in LINE_BREAKER after the () capturing group (space between events), that shows a new event is starting
.*? -- get the rest of the content for that event
((?=[\r\n]+\d\d\.\d\d\.\d\d\s)|(?!.)) -- this says collect content... until you find a spot that, after it, comes a newline followed by the above unique string. This means the event is ending. But, put an OR in there, because the last event won't have another event after it - so don't match anything.

Here is the full search string:

| rex field=_raw max_match=99 "(?sm)^(?<raw>\d\d\.\d\d\.\d\d\s.*?)((?=[\r\n]+\d\d\.\d\d\.\d\d\s)|(?!.))" | mvexpand raw | rename raw as _raw

Note: this only breaks up the raw text. Only up to 99 events will be extracted. Timestamps, field extractions, and event types will not be calculated from the "new" events.

View solution in original post

0 Karma

Jason
Motivator

Yes, you can use rex and multivalued fields to select new "raw" texts and expand them into new events. The fun is getting the regular expression right so you can use it like LINE_BREAKER. Here I am using a few things:

(?sm) -- s for "dot-all", so .*? matches newlines, and m for multiline, so ^ matches the beginning of a line instead of the beginning of an event
\d\d\.\d\d\.\d\d\s -- this would be the unique string, appearing in LINE_BREAKER after the () capturing group (space between events), that shows a new event is starting
.*? -- get the rest of the content for that event
((?=[\r\n]+\d\d\.\d\d\.\d\d\s)|(?!.)) -- this says collect content... until you find a spot that, after it, comes a newline followed by the above unique string. This means the event is ending. But, put an OR in there, because the last event won't have another event after it - so don't match anything.

Here is the full search string:

| rex field=_raw max_match=99 "(?sm)^(?<raw>\d\d\.\d\d\.\d\d\s.*?)((?=[\r\n]+\d\d\.\d\d\.\d\d\s)|(?!.))" | mvexpand raw | rename raw as _raw

Note: this only breaks up the raw text. Only up to 99 events will be extracted. Timestamps, field extractions, and event types will not be calculated from the "new" events.

0 Karma
Get Updates on the Splunk Community!

Splunk at Cisco Live 2025: Learning, Innovation, and a Little Bit of Mr. Brightside

Pack your bags (and maybe your dancing shoes)—Cisco Live is heading to San Diego, June 8–12, 2025, and Splunk ...

Splunk App Dev Community Updates – What’s New and What’s Next

Welcome to your go-to roundup of everything happening in the Splunk App Dev Community! Whether you're building ...

The Latest Cisco Integrations With Splunk Platform!

Join us for an exciting tech talk where we’ll explore the latest integrations in Cisco &#43; Splunk! We’ve ...