Multi-line event field extraction

[joshrabinowitz]

I have logs being indexed that look like:

/some/filesystem/path 1234567890 1500

/some/filesystem/path2 1256320145 452633

/some/filesystem2/path 2365800125 1122233

/some/filesystem2/path2 6539025630 553222

...

I am trying to get each line extracted into 3 fields (for example fs, fs_total, fs_used). Some logs will have 1 line, and some logs will have 400+ lines.

I have tried rex like

index=disk sourcetype="disk_logs"| rex field=_raw "(?i)(?m)[0-9]{1,15}(?<filesystem_used>.*)"

and that will give me the last column, but I'm not sure how to get all 3 columns into 3 fields for all the lines per log.

[ziegfried · Accepted solution]

Assuming that all lines are seperate events, you can use such a regex:

index=disk sourcetype="disk_logs" | rex "^(?<fs>.+?)\s+(?<fs_total>\d+)\s+(?<fs_used>\d+)$"

Otherwise you can split the events into seperate results:

index=disk sourcetype="disk_logs" | rex "(?m)^(?<line>\V+)$" max_match=10000 | table line | mvexpand line | rex field=line "^(?<fs>.+?)\s+(?<fs_total>\d+)\s+(?<fs_used>\d+)$"

[ziegfried]

Great! Please accept the answer.

[joshrabinowitz]

makes the perfect table, thank you so much! now i just need to figure out how to make some pretty graphs with each fs_total and fs_used as the y-axis and time as the x-axis (each event has a unix m-time which splunk seems to respect) and make a graph per fs

[ziegfried]

even if you did not generate the logs, you can still configure how splunk is doing the line breaking

transforms.conf

[disk_logs]
SHOULD_LINEMERGE = false

[ziegfried]

Is this by intention?

[joshrabinowitz]

works great! except for the events that have multiple lines. the largest is 442 lines in one event. not sure how to go about reading each line and extracting the 3 fields.