How to use splunk for binary log file?

[txing]

Our application is generating a binary format log files, and we want to use splunk to collect and search through it, can splunk process binary logs? what else should we do?

Thanks

[ziegfried]

Another option is to use the unarchive command. There has been a blog entry that illustrates how to do that:

http://blogs.splunk.com/2011/07/19/the-naughty-bits-how-to-splunk-binary-logfiles/

It still involves scripting, though.

[Ayn]

While you could technically force Splunk to index a file even though it's in a binary format, the question is what you'd get out of it. If your application logs in some kind of binary format where special knowledge is required to make any sense out of the data, you need to somehow provide that logic so that the data can be converted to something that makes sense to (and in) Splunk. A common way of doing this is using scripted inputs, where you have a script that reads your binary logs, retrieves the data and outputs it as a plaintext format that's suitable to be consumed by Splunk.

[ziegfried]

Creating such a scripted input is pretty simple. Everthing that the script sends to STDOUT will be indexed by Splunk.

[txing]

Thanks for the answers, but I'm not scripting guy, for the scripted input, is there a sample or we need to do it by ourselves?

Thanks

[MarioM]

some people like to look at hexadecimal... 😜

[MarioM]

you can allow processing of binary files in props.conf:

NO_BINARY_CHECK = [true|false]
* When set to true, Splunk processes binary files.
* Can only be used on the basis of [<sourcetype>], or [source::<source>], not [host::<host>].
* Defaults to false (binary files are ignored).