topic Re: Extracting fields from undelimited binary data? in All Apps and Add-ons

Extracting fields from undelimited binary data?

rapple1066 — Wed, 19 Jun 2013 19:22:46 GMT

I've got data coming in that's a hex string (binary fields). They're not delimited, but they do follow a fixed format.

Offset 0 , 1 byte = Index

Offset 1, 1 byte = Data Type

Offset 2, 2 bytes = Sequence Number

Offset 4, 4 bytes = Interval

Offset 8, 4 bytes = Timestamp (seconds)

Offset 12, 4 bytes = Timestamp2 (nanoseconds)

Offset 16, 4 bytes = 32 bit counter #1

Offset 20, 4 bytes = 32 bit counter #2

...followed by 30 additional 4 byte counter fields.

From what I understand, I need to use SEDCMD to insert delimiters and then use DELIM to allow the fields to be extracted? Any help on the syntax would be greatly appreciated since my SED is about 20 years rusty.

Re: Extracting fields from undelimited binary data?

cschmidt0121 — Mon, 28 Sep 2020 14:08:01 GMT

Couldn't you just do this with a rex extraction?

Something like:

rex field=_raw "(?.{2})(?.{2})(?.{4})(?.{8})(?.{8})(?.{8})(?.{8})(?.{8})... etc

Re: Extracting fields from undelimited binary data?

rapple1066 — Wed, 19 Jun 2013 21:27:56 GMT

I sure hope so... that looks vastly simpler than what I've been trying to do. I'll give that a shot.

Re: Extracting fields from undelimited binary data?

cschmidt0121 — Wed, 19 Jun 2013 21:38:56 GMT

If it doesn't, post an example of one of the raw events and I can try to fix my regex.

Re: Extracting fields from undelimited binary data?

Ayn — Thu, 20 Jun 2013 10:45:05 GMT

This blog post might be of interest, even though it's dealing with raw binary data and not just a hex representation of it: http://blogs.splunk.com/2011/07/19/the-naughty-bits-how-to-splunk-binary-logfiles/

Re: Extracting fields from undelimited binary data?

rapple1066 — Thu, 20 Jun 2013 20:58:07 GMT

Thanks all..

Here's a sample record:

00010cc503e851a8c733248e0b380274d41000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

This is how it shows up in Splunk:

\xA4\xFC\xE8Q\xC34,
\xA4\xFD\xE8Q\xC34,
\xA4\xFE\xE8Q\xC34,
\xA4\xFF\xE8Q\xC34,
\xA5
\xA5\xE8Q\xC34,
\xA5\xE8Q\xC34,
\xA5\xE8Q\xC34,
\xA5\xE8Q\xC34,

Re: Extracting fields from undelimited binary data?

cschmidt0121 — Thu, 20 Jun 2013 21:09:00 GMT

Is that how you want the data to look in splunk? If not I highly recommend setting up an input like the blog post Ayn suggested. Parse the data with a python script and output it as with human readable timestamps, fields, etc. To be honest, I have no clue how that Splunk excerpt could possibly represent the raw data.

Re: Extracting fields from undelimited binary data?

rapple1066 — Thu, 20 Jun 2013 21:25:00 GMT

Sorry.. that's how it shows up RAW in splunk when it comes in off the wire.

Maybe a better explanation of the data would help?

The data represents performance data (packet counts) from a network appliance. Every millisecond, we send a UDP packet to splunk that has the number of bytes observed in that time period. The beginning of the packet has some housekeeping info (Index, datatype, sequence #), 2 timestamps (seconds,and nanoseconds) and then the counter data from 32 "interfaces". The goal is to be able to report against each of the counters over time.

Re: Extracting fields from undelimited binary data?

rapple1066 — Thu, 20 Jun 2013 21:27:39 GMT

Here's what I'm seeing:

http://i236.photobucket.com/albums/ff31/spongerapple/splunk.jpg

Re: Extracting fields from undelimited binary data?

cschmidt0121 — Thu, 20 Jun 2013 21:42:27 GMT

Yeah, I definitely think the least painful solution to this is to simplify your data before it makes its way into Splunk. It looks like Splunk is trying and failing to parse the data - for example, isn't there a huge chunk of data missing? I count 36 bytes (minus all of the /x's) in each event in your screenshot. There should be a LOT more, correct?