Splunk Search

How to extract X number of fields based on another field value in the data?

wilsonite
Explorer

I am capturing some machine data and am wondering if it is possible to grab more or fewer fields via field extraction based on a size field in the data itself?

1480823739.999999 bus device [6] aa bb ff 00 33 33 
1480823741.999999 bus device [2] ab f0 

with the [6] and [2] in the data being the size values respectively.
My present field extraction regex looks like this:

^\s+\(\d+\.\d+\)\s+(?P<bus>\w{1,4})\s+(?P<device>\w{2,3})\s+\[(?P<data_len>\d)\]\s+

This gets me extracted up to the message bytes. While I can just import the data values as a single field, I would like to be able to pull each two hex characters into separate fields based on this size data.

Expanding on my regex, if I add multiple byte extractions to cover all instances, the smaller messages will not be extracted.

 ^\s+\(\d+\.\d+\)\s+(?P<bus>\w{1,4})\s+(?P<device>\w{2,3})\s+\[(?P<data_len>\d)\]\s+(?P<byte0>\w[0-9A-F]+)\s+(?P<byte1>\w[0-9A-F]+)\s+(?P<byte2...

This results in the 6 byte messages being trapped and none of the smaller messages will.

Can I have Splunk create multiple >byte1<,>byte2<,>byteN<... extractions based on the >data_len< field?

Thank you,
Wilsonite

0 Karma

sundareshr
Legend

What you could do is, create a multi-value field. In your props.conf add the following

EXTRACT-msg_bytes = \]\s(?<msg_bytes>.*)

This will extract all the message bytes into a mv field called msg_bytes. You can then use this in your search query to get to individual bits using split() or makemv.

base search that returns in msg_bytes amongst others | makemv msg_bytes delim=" " | eval msg_length=mvcount(msg_bytes) mvexpand msg_bytes | you should now have msg_bytes extract into individual events.

OR if you just want a specific one

base search that returns in msg_bytes amongst others | eval msg_bit=mvindex(split(msg_bytes, " "), 0) | this will give you the first bit etc.

wilsonite
Explorer

sundareshr, I cannot wait to try this tonight. Thank you for your insight!

0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee
0 Karma

wilsonite
Explorer

That is a good link as well, thank you dmaislin! Will let you folks know how it turns out.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...