topic Re: mainframe log parsing help - crazy log format in Splunk Search

mainframe log parsing help - crazy log format

ebailey — Sun, 23 Mar 2014 16:12:37 GMT

Greetings

I have been staring at the below for sometime and I have no idea where to start to get this log to parse correctly in Splunk. I would like to get the message broken down by message size and then parsed into 25 characters strings and then do a field extraction to break out all the fields I defined in the decoder ring. I am pretty sure I am approaching this wrong so any pointers are appreciated.

Thanks!

Here is the message and below is the decoder ring

"I cannot get the message to paste into the window correctly. The message is a single string with the message size "00987" and then 4 spaces before the rest of the message"

00987 201406919331234000930000020140691933123400067000002014069193311110007000000201406919331234000990000020140691933000000103000002014069193303370006300000201406919333150000090000020140691933600000002000002014069193362000000400000201406919336600000750000020140691933665000003000002014069193366700008100000201406919336672000670000020140691933668000009000002014069193366820000900000201406919336710000030000020140691933677000272000002014069193367900026100000201406919336792000090000020140691933700004425000102014069193370040004200000201406919337040002540000020140691933711100025000002014069193371120001300000201406919337200000060000020140691933730000006000002014069193373500016200000201406919337400002710000020140691933770000002000002014069193377600022600000201406919337770000050000020140691933791000004000002014069193379510001000000201406919338000001290000120140691933889000002000002014069193399000016400000201406919339910000140000020140691934651000001000002014069193361000003500000

Decoder Ring

00987 - message size

First 25 character block

2014069193312340009300000

Year (4) - 2014
Julian (3) - 069
24-hour time (4) - 1933
Product Code (4) - 1234
Success (5) - 00093
Errors (5) - 00000

Re: mainframe log parsing help - crazy log format

kristian_kolb — Mon, 24 Mar 2014 08:18:27 GMT

Hi, the following has been tested and works fine. The tricky part is to split the line into several events, which is done through the somewhat hard-to-read regex in LINE_BREAKER.

inputs.conf

[monitor:///path/to/your/mainframe.log]
sourcetype = my_mainframe

props.conf

[my_mainframe]
TIME_FORMAT = %Y%j%H%M
SHOULD_LINEMERGE = false
LINE_BREAKER = (?:\d+\s+)?(\B)(?=20[1-3]\d[0-3]\d\d[0-2]\d[0-5])
SEDCMD-dropfirstline = s/\d+\s+//
EXTRACT-blah = ^(?<my_date>\d{7})(?<my_time>\d{4})(?<product_code>\d{4})(?<success>\d{5})(?<errors>\d{5})

Hope this helps,

Re: mainframe log parsing help - crazy log format

ebailey — Mon, 24 Mar 2014 12:12:14 GMT

Nice - I will start testing - Thanks!

Re: mainframe log parsing help - crazy log format

kristian_kolb — Mon, 24 Mar 2014 12:50:51 GMT

Just be aware that the raw event looks like a large number in scientific notation, e.g.

2.014069e+24

This is just the way splunk presents large numbers. The field extractions will work just the same, and you can make your stats/top/table etc reporting on the extracted fields anyway.

Re: mainframe log parsing help - crazy log format

ebailey — Fri, 28 Mar 2014 04:48:48 GMT

Works great - I cannot figure out how to my_date to parse correctly but not a critical issue. - Thanks!

Re: mainframe log parsing help - crazy log format

kristian_kolb — Fri, 28 Mar 2014 07:05:03 GMT

Well, in the config example the my_date field is extracted as a string, but it is also (together the time portion), through the TIME_FORMAT extracted into the _time field, which can be used for further processing.

To extract and format the subparts, like the month, or day of the week, you can use the strftime() functions for eval, e.g.

... | eval my_special_date = strftime(_time,"%B:%d:%A-%H:%M")

which would give results like "March:28:Monday-16:34".

see www.strftime.net for a list of commonly used variables and their meaning.

Re: mainframe log parsing help - crazy log format

Tejkumar451 — Mon, 13 Feb 2017 19:26:21 GMT

Hi can you let us know how are you getting the mainframes data into splunk please

Re: mainframe log parsing help - crazy log format

ebaileytu — Wed, 01 Mar 2017 12:59:20 GMT

Mostly MF jobs that drop files into a ftp directory and then splunk logs in and downloads the logs from the ftp site. Also a Perl script that pulls data from a custom MF program buffer. That is the voodoo that produces the output this post is about. If you can afford it the solution from IronPort is awesome. It really works well and avoids the hell of trying to pull information from a MF by other means.