topic Re: Why does Splunk think my file is binary in Getting Data In

Why does Splunk think my file is binary

a212830 — Tue, 29 Sep 2020 08:56:22 GMT

Hi,

I'm trying to process a ".log" file on a Windows server, and Splunk keeps ignoring it, stating that it's a binary file.

02-26-2016 09:26:54.574 -0500 WARN  FileClassifierManager - The file 'C:\Temp\w32tmdebug.log' is invalid. Reason: binary
02-26-2016 09:26:54.574 -0500 INFO  TailReader - Ignoring file 'C:\Temp\w32tmdebug.log' due to: binary

I am able to open the file using notepad, so I'm not sure why Splunk thinks it's binary. I also tried adding "NO_BINARY_CHECK" but that didn't work either. My inputs is below. Any suggestions?

[monitor://C:\Temp\w32tmdebug.log]
disabled = false
followTail = 0
index = main
sourcetype = ntpdebug_log
ignoreOlderThan = 2d
NO_BINARY_CHECK = true

Re: Why does Splunk think my file is binary

JMichaelis — Fri, 26 Feb 2016 14:40:54 GMT

The missing backslash in your filepath could cause problems.

Re: Why does Splunk think my file is binary

skoelpin — Fri, 26 Feb 2016 14:58:59 GMT

What kind of data is in your log file? I know PDF files and such are treated as binary

Re: Why does Splunk think my file is binary

a212830 — Fri, 26 Feb 2016 15:03:34 GMT

it's text. I can open and edit the file with notepad. Windows says it's a text file when you look at it in the folder view.

Re: Why does Splunk think my file is binary

muebel — Tue, 29 Sep 2020 08:56:25 GMT

Hi a212830, the NO_BINARY_CHECK is a props.conf configuration, and so you will want to create a stanza in props like:

[source::C:\Temp\w32tmdebug.log]
NO_BINARY_CHECK = true

However, I'd take a closer look at this log file. The encoding is probably screwy and therefore is throwing off Splunk. Windows logs can be tricky like that.

Please let me know if this helps!

Re: Why does Splunk think my file is binary

javiergn — Fri, 26 Feb 2016 15:06:06 GMT

Is your file using something other than the UTF-8 or ASCII character sets? For instance, I had a similar problem with some logs encoded in UTF-16 and had to specify it explicitly.

See this:

http://docs.splunk.com/Documentation/Splunk/6.3.3/data/Configurecharactersetencoding#Comprehensive_list_of_supported_character_sets

Re: Why does Splunk think my file is binary

ddrillic — Tue, 29 Sep 2020 08:56:33 GMT

Interestingly, the Out-Of-The-Box props.conf has a number of NO_BINARY_CHECK = 1 within it, such as -

[WinNetMonMk]
KV_MODE = multi_WinNetMonMk
NO_BINARY_CHECK = 1
pulldown_type = 0

Re: Why does Splunk think my file is binary

ddrillic — Fri, 26 Feb 2016 23:46:01 GMT

http://stackoverflow.com/questions/35335128/grep-thinks-text-file-is-binary-but-it-isnt speaks about the issue.

Re: Why does Splunk think my file is binary

sloshburch — Sat, 27 Feb 2016 00:37:46 GMT

I love all the answers and ideas posted here. I think I've come across this in the past and the root cause was the same as other folks on this thread have posted.

Here's some details on what I remember I did to determine if encoding was the cause:

Create a copy of the file (so you can muck around with it without impacting the production version)
Create a new monitor stanza, same as the old, for the new copy of the file. Validate that it still shows as "binary" when Splunk goes for it. This is a base case to make sure we're able to reproduce the problem.
Open the copy in Notepad++. There's an Encoding menu item. I forget if you have to select text first so feel free to select all and then check the encoding menu to see what is currently selected.
Try toggling to UTF-8 or ANSI options and restarting Splunk to see if the file gets indexed (no longer recognized as binary)

I hope it turns out to be as simple as this. Crossing my fingers.

Re: Why does Splunk think my file is binary

druckerg — Fri, 24 Mar 2017 14:51:54 GMT

,I have had the same problems as described above. When I copied the file to notepad++, saw that the file was ANSI, converted it to UTF-8, copied the converted UTF-8 file back and splunk forwarder was able to read/watch the file. So, I clearly defined the problem but what is my solution? How do I get Splunk to read these files that are created as ANSI?

Re: Why does Splunk think my file is binary

sloshburch — Mon, 27 Mar 2017 12:21:11 GMT

http://docs.splunk.com/Documentation/Splunk/latest/Data/Configurecharactersetencoding

There's a few ANSI options so it looks like some internet research or trial and error testing to see which is right.

Re: Why does Splunk think my file is binary

sloshburch — Mon, 27 Mar 2017 12:22:42 GMT

Here's another approach for determining the file character set: http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Garbledevents

So in this case, FTP the file from Windows to a Unix system to use the file command as a way to determine the character set.

Re: Why does Splunk think my file is binary

sylim_splunk — Tue, 29 Sep 2020 18:01:31 GMT

Below worked for many cases ; Make sure to add it to props.conf on the server where monitor stanza is configured - mostly UF.
["source or sourcetype "]
NO_BINARY_CHECK=true
CHARSET=AUTO

If it doesn't work then enable DEBUG and open a support ticket along with a diag with DEBUG messages.
- Below are the categories to be set to DEBUG

./splunk set log-level FileClassifierManager -level DEBUG
./splunk set log-level FileClassifier -level DEBUG
./splunk set log-level WatchedFile -level DEBUG
./splunk set log-level TailingProcessor -level DEBUG
./splunk set log-level TailReader -level DEBUG

Re: Why does Splunk think my file is binary

abheeksett — Wed, 30 Sep 2020 02:25:45 GMT

yes CHARSET=AUTO worked for me .

So my inputs.conf

[monitor://C:\Windows\System32\LogFiles\SMTPSVC1]
disabled = false
index=smtp
sourcetype = ms:iis:auto
crcSalt =
initCrcLength = 512

props.conf

[ms:iis:auto]
TZ = Australia/Melbourne
NO_BINARY_CHECK = true
CHARSET=AUTO