Getting Data In

Line break with multiple Linebreaker

ss026381
Communicator

I have a log file something like this

PStart  bla bla bla bla 
PQL% bla bla bla bla
PEnd bla bla bla bla 
PQL% bla bla bla bla
bla bla blabla bla bla
PQL% bla bla bla bla

I want to break it at PStart or PQL% but if PQL% is within PStart and PEnd keep it one event.

For the above example, I want to have 3 events as follow

Event 1

PStart  bla bla bla bla 
PQL% bla bla bla bla
PEnd bla bla bla bla 

Event 2

PQL% bla bla bla bla
bla bla blabla bla bla

Event 3

PQL% bla bla bla bla

The code I have is following which doesn't work properly

[sourcetype]
LINE_BREAKER = ([\r\n]+)(?:(?:PStart|(?:PQL%))) 
MUST_NOT_BREAK_BEFORE = ([\r\n]+)(?:PEnd) 
Tags (2)
1 Solution

acharlieh
Influencer

With the sample as given this is what I come up with:

[sourcetype]
LINE_BREAKER=([\r\n]+)(?:PStart|PEnd|PQL%)
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE=^PStart|^PQL%
MUST_NOT_BREAK_AFTER=^PStart
MUST_BREAK_AFTER=^PEnd

First during the line breaking stage, we break up into "lines" that start PStart, PEnd or PQL% (This is only necessary if a PEnd line may contain carrage returns or line feeds, otherwise you can get away with the default ([\r\n]+) here

Then during line merging, we only break events before a PStart or PQL% line... this gets us our rough separation of events... but then with the MUST_NOT_BREAK_AFTER / MUST_BREAK_AFTER combination, we can ensure that from PStart to PEnd is a single block. (Obviously does not support embedding, but works single level).

View solution in original post

acharlieh
Influencer

With the sample as given this is what I come up with:

[sourcetype]
LINE_BREAKER=([\r\n]+)(?:PStart|PEnd|PQL%)
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE=^PStart|^PQL%
MUST_NOT_BREAK_AFTER=^PStart
MUST_BREAK_AFTER=^PEnd

First during the line breaking stage, we break up into "lines" that start PStart, PEnd or PQL% (This is only necessary if a PEnd line may contain carrage returns or line feeds, otherwise you can get away with the default ([\r\n]+) here

Then during line merging, we only break events before a PStart or PQL% line... this gets us our rough separation of events... but then with the MUST_NOT_BREAK_AFTER / MUST_BREAK_AFTER combination, we can ensure that from PStart to PEnd is a single block. (Obviously does not support embedding, but works single level).

ss026381
Communicator

This works, thanks...

0 Karma

woodcock
Esteemed Legend

Be sure to click Accept to close this question and reward the author of this MOST TERRIBLY EXCELLENT ANSWER!!!!

0 Karma

ss026381
Communicator

Thank you guys, I accepted, special thanks to Charlie.

0 Karma

woodcock
Esteemed Legend

This is a tough one. I am curious to see if there is a splunk-based solution. I've got nothing because negative-lookbehinds must be fixed-length.

0 Karma
Get Updates on the Splunk Community!

Dashboard Studio Challenge - Learn New Tricks, Showcase Your Skills, and Win Prizes!

Reimagine what you can do with your dashboards. Dashboard Studio is Splunk’s newest dashboard builder to ...

Introducing Edge Processor: Next Gen Data Transformation

We get it - not only can it take a lot of time, money and resources to get data into Splunk, but it also takes ...

Take the 2021 Splunk Career Survey for $50 in Amazon Cash

Help us learn about how Splunk has impacted your career by taking the 2021 Splunk Career Survey. Last year’s ...