Splunk Search

How to configure line breaking for mixed single line and multiline logs?

jamesvz84
Communicator

Given the following log format, is it possible to store the consecutive GROUPED/GROUPED_DET lines into one event while leaving LOGTYPE1 & LOGTYPE2 lines as singled-lined events? One of my main issues is also trying to come up with a regex for the LINE_BREAKER that will ONLY break if the line does NOT contain GROUPED : or GROUPED_DET:. Seems like Splunk will only look for a positive condition (break if x exists) rather than a negative condition (break if line does NOT contain x). Is that the case, or is there a regex or other settings that will allow proper breaking on this log?

LOGTYPE1 :2015/02/28 18:13:04.253|2015/02/28|118:13:04.252|6|LOC|4|0.01||||||||||||||0|50|41.25
LOGTYPE1 :2015/02/28 18:13:26.002|2015/02/28|18:13:26.002|5|GPS|3|1|39:48.0219|N|105:4.0201|W|2|10|1.0
GROUPED :2015/02/28 18:13:57.591:GROUP:Rcvd Office msg [1080] from AG
GROUPED_DET:2015/02/28 18:13:57.591:GROUP:000 | 04 04 38 01 11 00 00 09 00 03 d1 75 54 f2 05 66 | ..8........uT..f
GROUPED_DET:2015/02/28 18:13:57.592:GROUP:001 | 28 00 37 00 00 62 6e 73 66 2e 62 3a 70 74 63 73 | (.7..xxxx.b:ptcs
GROUPED_DET:2015/02/28 18:13:57.592:GROUP:002 | 72 76 72 00 62 6e 73 66 2e 6c 2e 62 6e 73 66 2e | rvr.xxxx.l.xxxx.
GROUPED_DET:2015/02/28 18:13:57.592:GROUP:003 | 34 34 31 37 3a 69 74 63 00 42 4e 53 46 01 00 00 | 4417:itc.XXXX...
GROUPED_DET:2015/02/28 18:13:57.592:GROUP:004 | 00 00 7d a3 6b f9                               | ..}.k.
LOGTYPE2 :2015/02/28 18:13:57.595|2015/02/28|18:13:57.594|21|EMP|2|1080|1|1|04043801110000090003d17554f205662800370000626e73662e623a7074637372767200626e73662e6c2e626e73662e343431373a69746300424e534601000000007da36bf9
GROUPED :2015/02/28 18:13:58.546:GROUP:Sending Office msg [2080]
GROUPED_DET:2015/02/28 18:13:58.546:GROUP:000 | 04 08 20 04 11 00 00 3b 00 03 d1 75 54 f2 05 66 | .. ....;...uT..f
GROUPED_DET:2015/02/28 18:13:58.546:GROUP:001 | 28 01 2c 00 20 62 6e 73 66 2e 6c 2e 62 6e 73 66 | (.,. xxxx.l.xxxx
GROUPED_DET:2015/02/28 18:13:58.546:GROUP:002 | 2e 34 34 31 37 3a 69 74 63 00 62 6e 73 66 2e 62 | .4417:itc.xxxx.b
GROUPED_DET:2015/02/28 18:13:58.546:GROUP:003 | 3a 70 74 63 73 72 76 72 00 42 4e 53 46 00 00 00 | :ptcsrvr.XXXX...
GROUPED_DET:2015/02/28 18:13:58.546:GROUP:004 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
GROUPED_DET:2015/02/28 18:13:58.547:GROUP:005 | 00 00 00 00 00 00 00 00 00 00 00 00 00 a4 00 f8 | ................
GROUPED_DET:2015/02/28 18:13:58.547:GROUP:006 | 65 2e 0d e3 c0 35 12 18 36 21 7b 02 00 02 04 01 | e....5..6!{.....
GROUPED_DET:2015/02/28 18:13:58.547:GROUP:007 | 00 00 00 00 cc da af de                         | ........
LOGTYPE1 :2015/02/28 18:13:58.550|2015/02/28|18:13:58.550|21|EMP|2|2080|1|1|040820041100003b0003d17554f2056628012c0020626e73662e6c2e626e73662e343431373a69746300626e73662e623a7074637372767200424e53460000000000000000000000000000000000000000000000000000000000000000a400f8652e0de3c035121836217b
0 Karma
1 Solution

jamesvz84
Communicator

The following worked (uses negative lookahead regex):

[sourcetype]
SHOULD_LINEMERGE = true 
LINE_BREAKER = ([\r\n]+)(?!GROUPED_DET)

View solution in original post

jamesvz84
Communicator

The following worked (uses negative lookahead regex):

[sourcetype]
SHOULD_LINEMERGE = true 
LINE_BREAKER = ([\r\n]+)(?!GROUPED_DET)
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...