Getting Data In

Why does Splunk think my file is binary

a212830
Champion

Hi,

I'm trying to process a ".log" file on a Windows server, and Splunk keeps ignoring it, stating that it's a binary file.

02-26-2016 09:26:54.574 -0500 WARN  FileClassifierManager - The file 'C:\Temp\w32tmdebug.log' is invalid. Reason: binary
02-26-2016 09:26:54.574 -0500 INFO  TailReader - Ignoring file 'C:\Temp\w32tmdebug.log' due to: binary

I am able to open the file using notepad, so I'm not sure why Splunk thinks it's binary. I also tried adding "NO_BINARY_CHECK" but that didn't work either. My inputs is below. Any suggestions?

[monitor://C:\Temp\w32tmdebug.log]
disabled = false
followTail = 0
index = main
sourcetype = ntpdebug_log
ignoreOlderThan = 2d
NO_BINARY_CHECK = true
1 Solution

sylim_splunk
Splunk Employee
Splunk Employee

Below worked for many cases ; Make sure to add it to props.conf on the server where monitor stanza is configured - mostly UF.
["source or sourcetype "]
NO_BINARY_CHECK=true
CHARSET=AUTO

If it doesn't work then enable DEBUG and open a support ticket along with a diag with DEBUG messages.
- Below are the categories to be set to DEBUG

./splunk set log-level FileClassifierManager -level DEBUG
./splunk set log-level FileClassifier -level DEBUG
./splunk set log-level WatchedFile -level DEBUG
./splunk set log-level TailingProcessor -level DEBUG
./splunk set log-level TailReader -level DEBUG

View solution in original post

sylim_splunk
Splunk Employee
Splunk Employee

Below worked for many cases ; Make sure to add it to props.conf on the server where monitor stanza is configured - mostly UF.
["source or sourcetype "]
NO_BINARY_CHECK=true
CHARSET=AUTO

If it doesn't work then enable DEBUG and open a support ticket along with a diag with DEBUG messages.
- Below are the categories to be set to DEBUG

./splunk set log-level FileClassifierManager -level DEBUG
./splunk set log-level FileClassifier -level DEBUG
./splunk set log-level WatchedFile -level DEBUG
./splunk set log-level TailingProcessor -level DEBUG
./splunk set log-level TailReader -level DEBUG

abheeksett
Engager

yes CHARSET=AUTO worked for me .

So my inputs.conf

[monitor://C:\Windows\System32\LogFiles\SMTPSVC1]
disabled = false
index=smtp
sourcetype = ms:iis:auto
crcSalt =
initCrcLength = 512

props.conf

[ms:iis:auto]
TZ = Australia/Melbourne
NO_BINARY_CHECK = true
CHARSET=AUTO

0 Karma

druckerg
New Member

,I have had the same problems as described above. When I copied the file to notepad++, saw that the file was ANSI, converted it to UTF-8, copied the converted UTF-8 file back and splunk forwarder was able to read/watch the file. So, I clearly defined the problem but what is my solution? How do I get Splunk to read these files that are created as ANSI?

0 Karma

sloshburch
Splunk Employee
Splunk Employee

http://docs.splunk.com/Documentation/Splunk/latest/Data/Configurecharactersetencoding

There's a few ANSI options so it looks like some internet research or trial and error testing to see which is right.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

I love all the answers and ideas posted here. I think I've come across this in the past and the root cause was the same as other folks on this thread have posted.

Here's some details on what I remember I did to determine if encoding was the cause:

  1. Create a copy of the file (so you can muck around with it without impacting the production version)
  2. Create a new monitor stanza, same as the old, for the new copy of the file. Validate that it still shows as "binary" when Splunk goes for it. This is a base case to make sure we're able to reproduce the problem.
  3. Open the copy in Notepad++. There's an Encoding menu item. I forget if you have to select text first so feel free to select all and then check the encoding menu to see what is currently selected.
  4. Try toggling to UTF-8 or ANSI options and restarting Splunk to see if the file gets indexed (no longer recognized as binary)

I hope it turns out to be as simple as this. Crossing my fingers.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Here's another approach for determining the file character set: http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Garbledevents

So in this case, FTP the file from Windows to a Unix system to use the file command as a way to determine the character set.

0 Karma

ddrillic
Ultra Champion
0 Karma

ddrillic
Ultra Champion

Interestingly, the Out-Of-The-Box props.conf has a number of NO_BINARY_CHECK = 1 within it, such as -

[WinNetMonMk]
KV_MODE = multi_WinNetMonMk
NO_BINARY_CHECK = 1
pulldown_type = 0

0 Karma

javiergn
SplunkTrust
SplunkTrust

Is your file using something other than the UTF-8 or ASCII character sets? For instance, I had a similar problem with some logs encoded in UTF-16 and had to specify it explicitly.

See this:

http://docs.splunk.com/Documentation/Splunk/6.3.3/data/Configurecharactersetencoding#Comprehensive_l...

muebel
SplunkTrust
SplunkTrust

Hi a212830, the NO_BINARY_CHECK is a props.conf configuration, and so you will want to create a stanza in props like:

[source::C:\Temp\w32tmdebug.log]
NO_BINARY_CHECK = true

However, I'd take a closer look at this log file. The encoding is probably screwy and therefore is throwing off Splunk. Windows logs can be tricky like that.

Please let me know if this helps!

skoelpin
SplunkTrust
SplunkTrust

What kind of data is in your log file? I know PDF files and such are treated as binary

0 Karma

a212830
Champion

it's text. I can open and edit the file with notepad. Windows says it's a text file when you look at it in the folder view.

0 Karma

JMichaelis
Path Finder

The missing backslash in your filepath could cause problems.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...