How to parse multi-line mixed messages from rsyslog?

[Rialf1959]

How to parse multi-line mixed messages from rsyslog?
There are a lot of data from lot of applications comming from Docker with syslog driver. All of these applications have proper syslog tag. Problem is that I cannot touch application code (Java). .... problem is multi-line stacktraces, these are flushed to docker stdout line by line. It means that rsyslog handle these lines as seperate messages.

I have tcp monitor in inputs.conf

What i see in splunk is:

<$pri>  $timestamp $host $syslogtag[$process]  $app_payload_msg

So for example:

<1> 2017-11-03T08:30:55.311256+01:00 myhost firstapp[1] 2017-11-03 08:31:55.31,301 ERROR  [myclass] Stacktrace1
<1> 2017-11-03T08:31:55.313649+01:00 myhost secondapp[1] 2017-11-03 08:31:55.31,301 WARN [myclass] message
<1> 2017-11-03T08:32:55.311256+01:00 myhost firstapp[1]      stacktraceline2
<1> 2017-11-03T08:33:55.311256+01:00 myhost firstapp[1]      stacktraceline3
<1> 2017-11-03T08:34:55.313649+01:00 myhost thirdapp[1] 2017-11-03 08:34:55.31,301 INFO [myclass] message
<1> 2017-11-03T08:35:55.311256+01:00 myhost firstapp[1]      stacktraceline4

I want these separate messsages group to events like:

  <1> 2017-11-03T08:30:55.311256+01:00 myhost firstapp[1] 2017-11-03 08:31:55.31,301 ERROR  [myclass] Stacktrace1
                             stacktraceline2
                             stacktraceline3
                             stacktraceline4
 <1> 2017-11-03T08:31:55.313649+01:00 myhost secondapp[1] 2017-11-03 08:31:55.31,301 WARN [myclass] message
 <1> 2017-11-03T08:34:55.313649+01:00 myhost thirdapp[1] 2017-11-03 08:34:55.31,301 INFO [myclass] message

Thanks

[skalliger]

Recently a college asked me the same thing; how he could handle stack traces in multi-line events.

Stack traces usually have the same format - where the following lines are starting with something like "[...] caused by. [..]",
so I would write a RegEx which does a BREAK_ONLY_BEFORE. After that , you simply define a) your RegEx and b) an "OR" after your RegEx to do the event breaking after a new line feed (\n), something like this (just an example):

((?=stacktraceline|\n)

Just add a stanza with your sourcetype in your transforms.conf:

[your_sourcetype]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = ((?=stacktraceline\d)|\n)

Skalli

edit: typo

[outcoldman]

Not an answer on your question, but an alternative, we have build a collector and Monitoring Docker application, which handles multiline events as well. Certified Splunk application https://splunkbase.splunk.com/app/3723/