splunking hex-based log events

highiqboy — Sat, 02 Oct 2010 23:16:41 GMT

I have log files from a custom app we wrote that is entirely in hex.

To splunk it, I understand I might be able to create a custom command that converts hex-to-ascii and then pipe to it at search time and then pipe again to "search some ascii terms"

Could I also, though, create a custom hex-to-ascii module or component and insert it into pipeline.xml after input step and before the indexing step? I believe that approach was supported in Splunk v2.x or maybe it was v3.x.

Also, does that component need to be written in C/C++ or can it be a script instead?

Re: splunking hex-based log events

gkanapathy — Sun, 03 Oct 2010 22:54:14 GMT

You can not do the conversion at search time. Data presented to Splunk at index time must be text data, as Splunk fundamentally indexes text.

There is currently (4.1.5) no support for creating your own pipeline to insert between the file monitor and the rest of the Splunk indexing queue. The recommended solution currently is either:

Preprocess your binary data and write it to text files, and provide the files to Splunk via either the monitor or batch inputs.
Create your own scripted input that does whatever it needs to do to generate text output and writes it to standard output. It does not matter what this is written in. Splunk will simply call the program and index whatever comes from its standard output stream. If you are trying to convert files,

Unfortunately both solutions have the disadvantage that you will have to code all file-tracking logic on your own in your program, rather than being able to use the Splunk file input monitor to do this.

topic Re: splunking hex-based log events in Getting Data In

splunking hex-based log events

Re: splunking hex-based log events