topic Re: Multi-line event field extraction in Splunk Search

Multi-line event field extraction

joshrabinowitz — Wed, 15 Jun 2011 18:14:09 GMT

I have logs being indexed that look like:

/some/filesystem/path 1234567890 1500

/some/filesystem/path2 1256320145 452633

/some/filesystem2/path 2365800125 1122233

/some/filesystem2/path2 6539025630 553222

...

I am trying to get each line extracted into 3 fields (for example fs, fs_total, fs_used). Some logs will have 1 line, and some logs will have 400+ lines.

I have tried rex like

index=disk sourcetype="disk_logs"| rex field=_raw "(?i)(?m)[0-9]{1,15}(?<filesystem_used>.*)"

and that will give me the last column, but I'm not sure how to get all 3 columns into 3 fields for all the lines per log.

Re: Multi-line event field extraction

ziegfried — Wed, 15 Jun 2011 18:23:41 GMT

Assuming that all lines are seperate events, you can use such a regex:

index=disk sourcetype="disk_logs" | rex "^(?<fs>.+?)\s+(?<fs_total>\d+)\s+(?<fs_used>\d+)$"

Otherwise you can split the events into seperate results:

index=disk sourcetype="disk_logs" | rex "(?m)^(?<line>\V+)$" max_match=10000 | table line | mvexpand line | rex field=line "^(?<fs>.+?)\s+(?<fs_total>\d+)\s+(?<fs_used>\d+)$"

Re: Multi-line event field extraction

joshrabinowitz — Wed, 15 Jun 2011 18:29:03 GMT

works great! except for the events that have multiple lines. the largest is 442 lines in one event. not sure how to go about reading each line and extracting the 3 fields.

Re: Multi-line event field extraction

ziegfried — Wed, 15 Jun 2011 18:36:55 GMT

Is this by intention?

Re: Multi-line event field extraction

ziegfried — Wed, 15 Jun 2011 19:26:45 GMT

even if you did not generate the logs, you can still configure how splunk is doing the line breaking

transforms.conf

[disk_logs]
SHOULD_LINEMERGE = false

Re: Multi-line event field extraction

joshrabinowitz — Mon, 28 Sep 2020 09:40:44 GMT

makes the perfect table, thank you so much! now i just need to figure out how to make some pretty graphs with each fs_total and fs_used as the y-axis and time as the x-axis (each event has a unix m-time which splunk seems to respect) and make a graph per fs

Re: Multi-line event field extraction

ziegfried — Wed, 15 Jun 2011 23:50:01 GMT

Great! Please accept the answer.