How to extract specific lines in a multiline event based on regex match?

[popdeluxe]

I am trying to analyze exception logging that is written across multiple lines, and extract only certain lines of the event into fields. I have been reading documentation and posts which seem to suggest defining stanzas in transforms.conf and props.conf would be the preferred way to tackle this. I have tried to implement to no avail and am lost! I would appreciate any ideas/suggestions on how to properly implement!!

Here are a few example log snippets to help demonstrate the challenge. Given the following, I need to extract...

(a) exception message (in bold/italics)
(b) the first calling method from either SOURCE_B or SOURCE_C, but not SOURCE_A

(1)
20160825 12:51:16 unhandled error from dispatcher, sender:System.Windows.Threading.Dispatcher
System.NullReferenceException: Object reference not set to an instance of an object.
at SOURCE_A.Method(Object sender, ExecutedRoutedEventArgs e)
at SOURCE_A.Method(Object sender, ExecutedRoutedEventArgs e)
at SOURCE_B.Method(Object sender, ExecutedRoutedEventArgs e, CommandBinding commandBinding)

(2)
20160825 12:53:16 unhandled error from dispatcher, sender:System.Windows.Threading.Dispatcher
System.Runtime.InteropServices.COMException ().
at SOURCE_C.Method(FORMATETC& format, STGMEDIUM& medium)

with the following results

(1)
UE_msg: System.NullReferenceException: Object reference not set to an instance of an object.
UE_method: SOURCE_B.Method(Object sender, ExecutedRoutedEventArgs e, CommandBinding commandBinding)

(2)
UE_msg: System.Runtime.InteropServices.COMException ()
UE_method: SOURCE_C.Method(FORMATETC& format, STGMEDIUM& medium)

The logging is not very structured...but all of these exceptions include the "unhandled error" string pattern, with the high-level "exception message" following on the next line that I need to extract, then SOURCE_B or SOURCE_C methods following below that somewhere in the stacktrace. So my thoughts are to define a REGEX stanza in transforms.config as follows

(transforms.config)

[UE_regex]
REGEX = (?m)(unhandled error.\*\n)(.\*\\.)((SOURCE_B|SOURCE_C).*\\))
FORMAT = UE_msg::$2  UE_method::$3

(props.config)

[UE]
REPORT-UE = UE_regex

lastly...try to table results....

source="c:\\logs\\perf*" sourcetype="UE" | table results

I have been tweaking the REGEX patterns, groupings, tried ditching transforms.config and tried defining just an EXTRACT in props.config. But nothing has yielded any results. At this point I can't tell if I'm even on the correct path anymore and would appreciate some guidance!

thanks!!!

[sundareshr · Accepted solution]

Try separate REPORT. Like this

*props*

[UE]
REPORT-UE = UE_msg
REPORT-UE = UE_method

*transforms*

[UE_msg]
REGEX = (?ms)unhandled error.*\n(?<UE_msg>[^\n]+)

[UE_method]
REGEX = at\s+(?<UE_method>SOURCE_[B|C])

[popdeluxe]

thank you sundareshr - this got me on the right track.

I tried the separate report, but it was only taking the last-assign stanza. i.e. UE_method was extracted but not UE_msg.

Continuing with this approach however, the following seems to work:

props.conf

REPORT-UEmsg = UE_msg
REPORT-UEmethod = UE_method

transforms.conf

[UE_msg]
REGEX = (?m)unhandled error.*(?System\D+\:.*)\.\s

[UE_method]
REGEX = (?m)unhandled error.*(?(SOURCE_B|SOURCE_C)\D+)\s

this has gotten me very close to what I need. I still am wrestling with the regex of the UE_method as it is extracting the rest of the stack trace instead of the specific line I want, but I will other posts for a solution and post a new one if needed.

thanks!