topic Re: Extract field from a complex multi-lines event from log file in Splunk Search

Extract field from a complex multi-lines event from log file

langlv — Tue, 31 May 2016 02:58:26 GMT

Hi pros,
I am new with Splunk and trying to analyze a complex log file from a financial application. I want to figure out the fields from a multilines event, here is my log example:

16.02.10 09:20:53 [ FromIso:123456789]************** INBOUND MESSAGE ID[AAABqgAwV0ujhQAA] ***************
 in[ 48: ]<800>
 in[ 48: ]<8220000100000000>
 in[ 48: ]<0400000000000000>
msgno[  0]<800>
Bitmap: [82200001000000000400000000000000]
 in[  7: ]<530>
 in[  7: ]<92833>
 in[ 11: ]<694437>
 in[ 32: ]<6>
 in[ 32: ]<123123>
 in[ 70: ]<2>
D-ISO-0306: m0800/a0000000000/t694437/p000000/r00
16.02.10 09:20:53 [   ToIso:123456789]************** OUTBOUND MESSAGE ID[AAABqgAwV0ujhQAA] ***************
msgno[  0]<810>
Bitmap: [82200001020000000400000000000000]
out[ 48: ]<0810>
out[ 48: ]<8220000102000000>
out[ 48: ]<0400000000000000>
out[  7: ]<0530>
out[  7: ]<092833>
out[ 11: ]<694437>
out[ 32: ]<06>
out[ 32: ]<123123>
out[ 39: ]<00>
out[ 70: ]<002>

I want to extract the fields msgno=800/810 and field39="00" from the log above.

I tried to use Field Extractor to extract those fields but no luck.

Thanks,
Lang

Re: Extract field from a complex multi-lines event from log file

kristian_kolb — Tue, 31 May 2016 09:50:47 GMT

Probably the best way is to use EXTRACTs in props.conf with the following regexes;

msgno[^<]+(?<msgno>\d+)
\s39:\s\]\<(?<field39>\d+)

You should probably check the documentation regarding EXTRACT

http://docs.splunk.com/Documentation/Splunk/6.4.1/Knowledge/Createandmaintainsearch-timefieldextractionsthroughconfigurationfiles
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

Also, it may be worth checking out the rex command, which lets you perform regex-based field extractions as part of the search query. At least it is simpler for trying out new extraction patterns before committing them to a config file.

http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Rex

Re: Extract field from a complex multi-lines event from log file

woodcock — Tue, 31 May 2016 13:18:51 GMT

At search time, like this:

... | rex max_match=99 "msgno\[\s*\d+\]<(?<msgno>\d+)>"
    | rex max_match=99 "out\[\s*39:\s*\]<(?<field39>\d+)>"

Re: Extract field from a complex multi-lines event from log file

langlv — Wed, 01 Jun 2016 04:27:38 GMT

It works like a charm.
Thanks Woodcock,