topic Re: How to extract X number of fields based on another field value in the data? in Splunk Search

How to extract X number of fields based on another field value in the data?

wilsonite — Sun, 04 Dec 2016 04:26:16 GMT

I am capturing some machine data and am wondering if it is possible to grab more or fewer fields via field extraction based on a size field in the data itself?

1480823739.999999 bus device [6] aa bb ff 00 33 33 
1480823741.999999 bus device [2] ab f0

with the [6] and [2] in the data being the size values respectively.
My present field extraction regex looks like this:

^\s+\(\d+\.\d+\)\s+(?P<bus>\w{1,4})\s+(?P<device>\w{2,3})\s+\[(?P<data_len>\d)\]\s+

This gets me extracted up to the message bytes. While I can just import the data values as a single field, I would like to be able to pull each two hex characters into separate fields based on this size data.

Expanding on my regex, if I add multiple byte extractions to cover all instances, the smaller messages will not be extracted.

 ^\s+\(\d+\.\d+\)\s+(?P<bus>\w{1,4})\s+(?P<device>\w{2,3})\s+\[(?P<data_len>\d)\]\s+(?P<byte0>\w[0-9A-F]+)\s+(?P<byte1>\w[0-9A-F]+)\s+(?P<byte2...

This results in the 6 byte messages being trapped and none of the smaller messages will.

Can I have Splunk create multiple >byte1<,>byte2<,>byteN<... extractions based on the >data_len< field?

Thank you,
Wilsonite

Re: How to extract X number of fields based on another field value in the data?

dmaislin_splunk — Sun, 04 Dec 2016 15:50:51 GMT

Read this. I think this will help: https://answers.splunk.com/answers/103700/how-do-i-create-a-field-whose-name-is-the-value-of-another-field-like-backticks-or-eval-in-other-languages.html

Re: How to extract X number of fields based on another field value in the data?

sundareshr — Sun, 04 Dec 2016 15:57:03 GMT

What you could do is, create a multi-value field. In your props.conf add the following

EXTRACT-msg_bytes = \]\s(?<msg_bytes>.*)

This will extract all the message bytes into a mv field called msg_bytes. You can then use this in your search query to get to individual bits using split() or makemv.

base search that returns in msg_bytes amongst others | makemv msg_bytes delim=" " | eval msg_length=mvcount(msg_bytes) mvexpand msg_bytes | you should now have msg_bytes extract into individual events.

OR if you just want a specific one

base search that returns in msg_bytes amongst others | eval msg_bit=mvindex(split(msg_bytes, " "), 0) | this will give you the first bit etc.

Re: How to extract X number of fields based on another field value in the data?

wilsonite — Mon, 05 Dec 2016 15:49:43 GMT

sundareshr, I cannot wait to try this tonight. Thank you for your insight!

Re: How to extract X number of fields based on another field value in the data?

wilsonite — Mon, 05 Dec 2016 18:15:04 GMT

That is a good link as well, thank you dmaislin! Will let you folks know how it turns out.