Splunk Search

How to configure line breaking for mixed single line and multiline logs?

jamesvz84
Communicator

Given the following log format, is it possible to store the consecutive GROUPED/GROUPED_DET lines into one event while leaving LOGTYPE1 & LOGTYPE2 lines as singled-lined events? One of my main issues is also trying to come up with a regex for the LINE_BREAKER that will ONLY break if the line does NOT contain GROUPED : or GROUPED_DET:. Seems like Splunk will only look for a positive condition (break if x exists) rather than a negative condition (break if line does NOT contain x). Is that the case, or is there a regex or other settings that will allow proper breaking on this log?

LOGTYPE1 :2015/02/28 18:13:04.253|2015/02/28|118:13:04.252|6|LOC|4|0.01||||||||||||||0|50|41.25
LOGTYPE1 :2015/02/28 18:13:26.002|2015/02/28|18:13:26.002|5|GPS|3|1|39:48.0219|N|105:4.0201|W|2|10|1.0
GROUPED :2015/02/28 18:13:57.591:GROUP:Rcvd Office msg [1080] from AG
GROUPED_DET:2015/02/28 18:13:57.591:GROUP:000 | 04 04 38 01 11 00 00 09 00 03 d1 75 54 f2 05 66 | ..8........uT..f
GROUPED_DET:2015/02/28 18:13:57.592:GROUP:001 | 28 00 37 00 00 62 6e 73 66 2e 62 3a 70 74 63 73 | (.7..xxxx.b:ptcs
GROUPED_DET:2015/02/28 18:13:57.592:GROUP:002 | 72 76 72 00 62 6e 73 66 2e 6c 2e 62 6e 73 66 2e | rvr.xxxx.l.xxxx.
GROUPED_DET:2015/02/28 18:13:57.592:GROUP:003 | 34 34 31 37 3a 69 74 63 00 42 4e 53 46 01 00 00 | 4417:itc.XXXX...
GROUPED_DET:2015/02/28 18:13:57.592:GROUP:004 | 00 00 7d a3 6b f9                               | ..}.k.
LOGTYPE2 :2015/02/28 18:13:57.595|2015/02/28|18:13:57.594|21|EMP|2|1080|1|1|04043801110000090003d17554f205662800370000626e73662e623a7074637372767200626e73662e6c2e626e73662e343431373a69746300424e534601000000007da36bf9
GROUPED :2015/02/28 18:13:58.546:GROUP:Sending Office msg [2080]
GROUPED_DET:2015/02/28 18:13:58.546:GROUP:000 | 04 08 20 04 11 00 00 3b 00 03 d1 75 54 f2 05 66 | .. ....;...uT..f
GROUPED_DET:2015/02/28 18:13:58.546:GROUP:001 | 28 01 2c 00 20 62 6e 73 66 2e 6c 2e 62 6e 73 66 | (.,. xxxx.l.xxxx
GROUPED_DET:2015/02/28 18:13:58.546:GROUP:002 | 2e 34 34 31 37 3a 69 74 63 00 62 6e 73 66 2e 62 | .4417:itc.xxxx.b
GROUPED_DET:2015/02/28 18:13:58.546:GROUP:003 | 3a 70 74 63 73 72 76 72 00 42 4e 53 46 00 00 00 | :ptcsrvr.XXXX...
GROUPED_DET:2015/02/28 18:13:58.546:GROUP:004 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
GROUPED_DET:2015/02/28 18:13:58.547:GROUP:005 | 00 00 00 00 00 00 00 00 00 00 00 00 00 a4 00 f8 | ................
GROUPED_DET:2015/02/28 18:13:58.547:GROUP:006 | 65 2e 0d e3 c0 35 12 18 36 21 7b 02 00 02 04 01 | e....5..6!{.....
GROUPED_DET:2015/02/28 18:13:58.547:GROUP:007 | 00 00 00 00 cc da af de                         | ........
LOGTYPE1 :2015/02/28 18:13:58.550|2015/02/28|18:13:58.550|21|EMP|2|2080|1|1|040820041100003b0003d17554f2056628012c0020626e73662e6c2e626e73662e343431373a69746300626e73662e623a7074637372767200424e53460000000000000000000000000000000000000000000000000000000000000000a400f8652e0de3c035121836217b
0 Karma
1 Solution

jamesvz84
Communicator

The following worked (uses negative lookahead regex):

[sourcetype]
SHOULD_LINEMERGE = true 
LINE_BREAKER = ([\r\n]+)(?!GROUPED_DET)

View solution in original post

jamesvz84
Communicator

The following worked (uses negative lookahead regex):

[sourcetype]
SHOULD_LINEMERGE = true 
LINE_BREAKER = ([\r\n]+)(?!GROUPED_DET)
Get Updates on the Splunk Community!

Splunk Mobile: Your Brand-New Home Screen

Meet Your New Mobile Hub  Hello Splunk Community!  Staying connected to your data—no matter where you are—is ...

Introducing Value Insights (Beta): Understand the Business Impact your organization ...

Real progress on your strategic priorities starts with knowing the business outcomes your teams are delivering ...

Enterprise Security (ES) Essentials 8.3 is Now GA — Smarter Detections, Faster ...

As of today, Enterprise Security (ES) Essentials 8.3 is now generally available, helping SOC teams simplify ...