How to index a binary data from UDP?

[matthewgao]

I am trying to receive a binary data from UDP in splunk, I have tested many method to achieve it, but it still doesn't work.
I already add a NO_BINARY_CHECK = true in props.conf.

Is there anyone have experience on this?

[lphirke]

Splunk can not consume binary data, you have to convert it to a splunk readable format before forwarding it to splunk. are you trying to forward netflows to splunk?

[lphirke]

Try out IPFIXify
http://www.plixer.com/Products/ipfixify.html

[matthewgao]

I am trying to forward ipfix to splunk

[Damien_Dallimor · Accepted solution]

You could write a simple python modular input, that allows users to open up any UDP port, captures the UDP datagram , programmatically decodes the binary data into some textual format, transforms this into events in a best practice logging semantic, and writes these events out to Splunk over STDOUT.

[Damien_Dallimor]

Exactly 🙂

[matthewgao]

So you mean that I create a UDP listener by myself, Not use the splunkd to listen the UDP

[Damien_Dallimor]

You missed the point.

You program your own Modular Input listening on it's own UDP port.Think of it like a Splunk UDP proxy. So it can capture and preprocess any type of data(ie: raw bninary) into an appropriate format for Splunk because you are programming it.

[matthewgao]

Can it capture the RAW binary data? Splunk seems automatically ignore the binary data.

[alacercogitatus]

Splunk will consume any Human-Readable text, so Binary data is not going to work. You will most likely want to capture that data via UDP and write it to disk in ASCII, and then have Splunk consume the text.

[alacercogitatus]

Did this help you?