Splunk Search

mainframe log parsing help - crazy log format

ebailey
Communicator

Greetings

I have been staring at the below for sometime and I have no idea where to start to get this log to parse correctly in Splunk. I would like to get the message broken down by message size and then parsed into 25 characters strings and then do a field extraction to break out all the fields I defined in the decoder ring. I am pretty sure I am approaching this wrong so any pointers are appreciated.

Thanks!

Here is the message and below is the decoder ring

"I cannot get the message to paste into the window correctly. The message is a single string with the message size "00987" and then 4 spaces before the rest of the message"

00987 201406919331234000930000020140691933123400067000002014069193311110007000000201406919331234000990000020140691933000000103000002014069193303370006300000201406919333150000090000020140691933600000002000002014069193362000000400000201406919336600000750000020140691933665000003000002014069193366700008100000201406919336672000670000020140691933668000009000002014069193366820000900000201406919336710000030000020140691933677000272000002014069193367900026100000201406919336792000090000020140691933700004425000102014069193370040004200000201406919337040002540000020140691933711100025000002014069193371120001300000201406919337200000060000020140691933730000006000002014069193373500016200000201406919337400002710000020140691933770000002000002014069193377600022600000201406919337770000050000020140691933791000004000002014069193379510001000000201406919338000001290000120140691933889000002000002014069193399000016400000201406919339910000140000020140691934651000001000002014069193361000003500000

Decoder Ring

00987 - message size

First 25 character block

2014069193312340009300000

Year (4) - 2014
Julian (3) - 069
24-hour time (4) - 1933
Product Code (4) - 1234
Success (5) - 00093
Errors (5) - 00000

Tags (3)
0 Karma
1 Solution

kristian_kolb
Ultra Champion

Hi, the following has been tested and works fine. The tricky part is to split the line into several events, which is done through the somewhat hard-to-read regex in LINE_BREAKER.

inputs.conf

[monitor:///path/to/your/mainframe.log]
sourcetype = my_mainframe

props.conf

[my_mainframe]
TIME_FORMAT = %Y%j%H%M
SHOULD_LINEMERGE = false
LINE_BREAKER = (?:\d+\s+)?(\B)(?=20[1-3]\d[0-3]\d\d[0-2]\d[0-5])
SEDCMD-dropfirstline = s/\d+\s+//
EXTRACT-blah = ^(?<my_date>\d{7})(?<my_time>\d{4})(?<product_code>\d{4})(?<success>\d{5})(?<errors>\d{5})

Hope this helps,

K

View solution in original post

Tejkumar451
Explorer

Hi can you let us know how are you getting the mainframes data into splunk please

0 Karma

ebaileytu
Communicator

Mostly MF jobs that drop files into a ftp directory and then splunk logs in and downloads the logs from the ftp site. Also a Perl script that pulls data from a custom MF program buffer. That is the voodoo that produces the output this post is about. If you can afford it the solution from IronPort is awesome. It really works well and avoids the hell of trying to pull information from a MF by other means.

0 Karma

kristian_kolb
Ultra Champion

Hi, the following has been tested and works fine. The tricky part is to split the line into several events, which is done through the somewhat hard-to-read regex in LINE_BREAKER.

inputs.conf

[monitor:///path/to/your/mainframe.log]
sourcetype = my_mainframe

props.conf

[my_mainframe]
TIME_FORMAT = %Y%j%H%M
SHOULD_LINEMERGE = false
LINE_BREAKER = (?:\d+\s+)?(\B)(?=20[1-3]\d[0-3]\d\d[0-2]\d[0-5])
SEDCMD-dropfirstline = s/\d+\s+//
EXTRACT-blah = ^(?<my_date>\d{7})(?<my_time>\d{4})(?<product_code>\d{4})(?<success>\d{5})(?<errors>\d{5})

Hope this helps,

K

kristian_kolb
Ultra Champion

Well, in the config example the my_date field is extracted as a string, but it is also (together the time portion), through the TIME_FORMAT extracted into the _time field, which can be used for further processing.

To extract and format the subparts, like the month, or day of the week, you can use the strftime() functions for eval, e.g.

... | eval my_special_date = strftime(_time,"%B:%d:%A-%H:%M")

which would give results like "March:28:Monday-16:34".

see www.strftime.net for a list of commonly used variables and their meaning.

0 Karma

ebailey
Communicator

Works great - I cannot figure out how to my_date to parse correctly but not a critical issue. - Thanks!

0 Karma

kristian_kolb
Ultra Champion

Just be aware that the raw event looks like a large number in scientific notation, e.g.

2.014069e+24

This is just the way splunk presents large numbers. The field extractions will work just the same, and you can make your stats/top/table etc reporting on the extracted fields anyway.

/k

0 Karma

ebailey
Communicator

Nice - I will start testing - Thanks!

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...