Splunk Search

unable to process binary log file

clearslide_cwon
New Member

i have splunkforwarder running but once a while we run into issue with the following error about file being binary -

11-30-2015 03:28:02.240 -0800 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/var/log/tomcat/catalina.out'.
11-30-2015 03:28:07.418 -0800 WARN  FileClassifierManager - The file '/var/log/tomcat/catalina.out' is invalid. Reason: binary
11-30-2015 03:28:07.418 -0800 INFO  TailReader - Ignoring file '/var/log/tomcat/catalina.out' due to: binary

i have tried following the instruction here but didnt seem to work -
https://answers.splunk.com/answers/36739/how-can-we-monitor-binary-log-data-in-splunk-is-invalid-rea...

not sure if this is the file to edit though -

/opt/splunkforwarder/etc/system/default/props.conf

[catalina]
# 20151124 cwong - added NO_BINARY_CHECK = 1
NO_BINARY_CHECK = 1
Tags (1)
0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

Do NOT edit files in system/default - these will get overwritten on upgrades of Splunk.
Instead, add a props.conf to the app that contains your inputs.conf and make the change there. Use the local subdirectory if the app wasn't created by your team.
http://docs.splunk.com/Documentation/Splunk/6.3.1511/Admin/Configurationfiledirectories

View solution in original post

yannK
Splunk Employee
Splunk Employee

just to put my 2 cents.

try to add NO_BINARY_CHECK = 1
to the sourcetype in .../local/props.conf on the forwarder and indexer.
and in the .../local/inputs.conf on the forwarder under the monitor stanza
then restart to apply

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Do NOT edit files in system/default - these will get overwritten on upgrades of Splunk.
Instead, add a props.conf to the app that contains your inputs.conf and make the change there. Use the local subdirectory if the app wasn't created by your team.
http://docs.splunk.com/Documentation/Splunk/6.3.1511/Admin/Configurationfiledirectories

clearslide_cwon
New Member

hey martin,

thanks for the quick reply.
i added props.conf to the local/ dir -

cat /opt/splunkforwarder/etc/system/local/props.conf

[catalina]

20151124 cwong - added NO_BINARY_CHECK = 1

NO_BINARY_CHECK = 1
BREAK_ONLY_BEFORE_DATE = true
SHOULD_LINEMERGE=true
MAX_TIMESTAMP_LOOKAHEAD=30
TIME_PREFIX = ^
pulldown_type = 1
category = Application
description = Output produced by Apache Tomcat Catalina (System.out and System.err)

i still see the "WARN FileClassifierManager - The file '/var/log/tomcat/catalina.out' is invalid. Reason: binary" in splunkd.log

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Don't forget to set this on the indexer, timestamp extraction and the like is not handled by a universal forwarder.
Also restart instances after these changes.

0 Karma

clearslide_cwon
New Member

thanks martin, sorry for the slow reply. having the following worked:

/opt/splunkforwarder/etc/system/local/inputs.conf

[monitor:///var/log/tomcat/catalina.out]
sourcetype=tomcat-catalina-out

/opt/splunkforwarder/etc/system/local/props.conf

[tomcat-catalina-out]
NO_BINARY_CHECK = true
BREAK_ONLY_BEFORE_DATE = true
SHOULD_LINEMERGE=true
MAX_TIMESTAMP_LOOKAHEAD=30
TIME_PREFIX = ^
pulldown_type = 1
category = Application
description = Output produced by Apache Tomcat Catalina (System.out and System.err)
0 Karma
Get Updates on the Splunk Community!

Introducing Ingest Actions: Filter, Mask, Route, Repeat

WATCH NOW Ingest Actions (IA) is the best new way to easily filter, mask and route your data in Splunk® ...

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...