Getting Data In

Line break with multiple Linebreaker

ss026381
Communicator

I have a log file something like this

PStart  bla bla bla bla 
PQL% bla bla bla bla
PEnd bla bla bla bla 
PQL% bla bla bla bla
bla bla blabla bla bla
PQL% bla bla bla bla

I want to break it at PStart or PQL% but if PQL% is within PStart and PEnd keep it one event.

For the above example, I want to have 3 events as follow

Event 1

PStart  bla bla bla bla 
PQL% bla bla bla bla
PEnd bla bla bla bla 

Event 2

PQL% bla bla bla bla
bla bla blabla bla bla

Event 3

PQL% bla bla bla bla

The code I have is following which doesn't work properly

[sourcetype]
LINE_BREAKER = ([\r\n]+)(?:(?:PStart|(?:PQL%))) 
MUST_NOT_BREAK_BEFORE = ([\r\n]+)(?:PEnd) 
Tags (2)
1 Solution

acharlieh
Influencer

With the sample as given this is what I come up with:

[sourcetype]
LINE_BREAKER=([\r\n]+)(?:PStart|PEnd|PQL%)
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE=^PStart|^PQL%
MUST_NOT_BREAK_AFTER=^PStart
MUST_BREAK_AFTER=^PEnd

First during the line breaking stage, we break up into "lines" that start PStart, PEnd or PQL% (This is only necessary if a PEnd line may contain carrage returns or line feeds, otherwise you can get away with the default ([\r\n]+) here

Then during line merging, we only break events before a PStart or PQL% line... this gets us our rough separation of events... but then with the MUST_NOT_BREAK_AFTER / MUST_BREAK_AFTER combination, we can ensure that from PStart to PEnd is a single block. (Obviously does not support embedding, but works single level).

View solution in original post

acharlieh
Influencer

With the sample as given this is what I come up with:

[sourcetype]
LINE_BREAKER=([\r\n]+)(?:PStart|PEnd|PQL%)
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE=^PStart|^PQL%
MUST_NOT_BREAK_AFTER=^PStart
MUST_BREAK_AFTER=^PEnd

First during the line breaking stage, we break up into "lines" that start PStart, PEnd or PQL% (This is only necessary if a PEnd line may contain carrage returns or line feeds, otherwise you can get away with the default ([\r\n]+) here

Then during line merging, we only break events before a PStart or PQL% line... this gets us our rough separation of events... but then with the MUST_NOT_BREAK_AFTER / MUST_BREAK_AFTER combination, we can ensure that from PStart to PEnd is a single block. (Obviously does not support embedding, but works single level).

ss026381
Communicator

This works, thanks...

0 Karma

woodcock
Esteemed Legend

Be sure to click Accept to close this question and reward the author of this MOST TERRIBLY EXCELLENT ANSWER!!!!

0 Karma

ss026381
Communicator

Thank you guys, I accepted, special thanks to Charlie.

0 Karma

woodcock
Esteemed Legend

This is a tough one. I am curious to see if there is a splunk-based solution. I've got nothing because negative-lookbehinds must be fixed-length.

0 Karma
Get Updates on the Splunk Community!

Splunk Security Content for Threat Detection & Response, Q1 Roundup

Join Principal Threat Researcher, Michael Haag, as he walks through:An introduction to the Splunk Threat ...

Splunk Life | Happy Pride Month!

Happy Pride Month, Splunk Community! 🌈 In the United States, as well as many countries around the ...

SplunkTrust | Where Are They Now - Michael Uschmann

The Background Five years ago, Splunk published several videos showcasing members of the SplunkTrust to share ...