Splunk Search

How do I simulate LINE_BREAKER at search time?

Jason
Motivator

I have some data that has been ingested quickly/badly, so there are multiple lines per event. Rather than reindex it, is there a way I can quickly break it up in a search, similar to LINE_BREAKER at parse time?

This is one event. I would like five:

20.30.40 this is an event -- The date is 6 January, 2017 --
20.30.41 this is another event
20.30.42 this is a multiline event?
  see?
  it's really a multiline event
  I promise.
20.30.43 this is a fourth event
20.30.44 and a final one
0 Karma
1 Solution

Jason
Motivator

Yes, you can use rex and multivalued fields to select new "raw" texts and expand them into new events. The fun is getting the regular expression right so you can use it like LINE_BREAKER. Here I am using a few things:

(?sm) -- s for "dot-all", so .*? matches newlines, and m for multiline, so ^ matches the beginning of a line instead of the beginning of an event
\d\d\.\d\d\.\d\d\s -- this would be the unique string, appearing in LINE_BREAKER after the () capturing group (space between events), that shows a new event is starting
.*? -- get the rest of the content for that event
((?=[\r\n]+\d\d\.\d\d\.\d\d\s)|(?!.)) -- this says collect content... until you find a spot that, after it, comes a newline followed by the above unique string. This means the event is ending. But, put an OR in there, because the last event won't have another event after it - so don't match anything.

Here is the full search string:

| rex field=_raw max_match=99 "(?sm)^(?<raw>\d\d\.\d\d\.\d\d\s.*?)((?=[\r\n]+\d\d\.\d\d\.\d\d\s)|(?!.))" | mvexpand raw | rename raw as _raw

Note: this only breaks up the raw text. Only up to 99 events will be extracted. Timestamps, field extractions, and event types will not be calculated from the "new" events.

View solution in original post

0 Karma

Jason
Motivator

Yes, you can use rex and multivalued fields to select new "raw" texts and expand them into new events. The fun is getting the regular expression right so you can use it like LINE_BREAKER. Here I am using a few things:

(?sm) -- s for "dot-all", so .*? matches newlines, and m for multiline, so ^ matches the beginning of a line instead of the beginning of an event
\d\d\.\d\d\.\d\d\s -- this would be the unique string, appearing in LINE_BREAKER after the () capturing group (space between events), that shows a new event is starting
.*? -- get the rest of the content for that event
((?=[\r\n]+\d\d\.\d\d\.\d\d\s)|(?!.)) -- this says collect content... until you find a spot that, after it, comes a newline followed by the above unique string. This means the event is ending. But, put an OR in there, because the last event won't have another event after it - so don't match anything.

Here is the full search string:

| rex field=_raw max_match=99 "(?sm)^(?<raw>\d\d\.\d\d\.\d\d\s.*?)((?=[\r\n]+\d\d\.\d\d\.\d\d\s)|(?!.))" | mvexpand raw | rename raw as _raw

Note: this only breaks up the raw text. Only up to 99 events will be extracted. Timestamps, field extractions, and event types will not be calculated from the "new" events.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...