Splunk Search

Multi-line event field extraction

joshrabinowitz
Path Finder

I have logs being indexed that look like:

/some/filesystem/path 1234567890 1500

/some/filesystem/path2 1256320145 452633

/some/filesystem2/path 2365800125 1122233

/some/filesystem2/path2 6539025630 553222

...

I am trying to get each line extracted into 3 fields (for example fs, fs_total, fs_used). Some logs will have 1 line, and some logs will have 400+ lines.

I have tried rex like

index=disk sourcetype="disk_logs"| rex field=_raw "(?i)(?m)[0-9]{1,15}(?<filesystem_used>.*)"

and that will give me the last column, but I'm not sure how to get all 3 columns into 3 fields for all the lines per log.

Tags (2)
0 Karma
1 Solution

ziegfried
Influencer

Assuming that all lines are seperate events, you can use such a regex:

index=disk sourcetype="disk_logs" | rex "^(?<fs>.+?)\s+(?<fs_total>\d+)\s+(?<fs_used>\d+)$"

Otherwise you can split the events into seperate results:

index=disk sourcetype="disk_logs" | rex "(?m)^(?<line>\V+)$" max_match=10000 | table line | mvexpand line | rex field=line "^(?<fs>.+?)\s+(?<fs_total>\d+)\s+(?<fs_used>\d+)$"

View solution in original post

ziegfried
Influencer

Assuming that all lines are seperate events, you can use such a regex:

index=disk sourcetype="disk_logs" | rex "^(?<fs>.+?)\s+(?<fs_total>\d+)\s+(?<fs_used>\d+)$"

Otherwise you can split the events into seperate results:

index=disk sourcetype="disk_logs" | rex "(?m)^(?<line>\V+)$" max_match=10000 | table line | mvexpand line | rex field=line "^(?<fs>.+?)\s+(?<fs_total>\d+)\s+(?<fs_used>\d+)$"

ziegfried
Influencer

Great! Please accept the answer.

0 Karma

joshrabinowitz
Path Finder

makes the perfect table, thank you so much! now i just need to figure out how to make some pretty graphs with each fs_total and fs_used as the y-axis and time as the x-axis (each event has a unix m-time which splunk seems to respect) and make a graph per fs

0 Karma

ziegfried
Influencer

even if you did not generate the logs, you can still configure how splunk is doing the line breaking

transforms.conf

[disk_logs]
SHOULD_LINEMERGE = false
0 Karma

ziegfried
Influencer

Is this by intention?

0 Karma

joshrabinowitz
Path Finder

works great! except for the events that have multiple lines. the largest is 442 lines in one event. not sure how to go about reading each line and extracting the 3 fields.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...