Hi
We recently had a problem with one type of our indexed log files suddenly being recognized as binary.
This is the message we saw in splunkd.log:
WARN FileClassifierManager - Invalid file: /xy/mylog.log, reason: binary.
We don't know why this happened it is an xml log, the format may have changed this type of file did get indexed befor.
So we changed our props.conf on the indexer and added the following parameter for our sources:
NO_BINARY_CHECK = true
We already had this parameter previusly:
CHARSET = ISO-8859-2
We only saw very few events and a lot of warning messages:
WARN UTF8Processor - Using charset UTF-8 for events from 'source::xxx|host::xxx|remoteport::33270', as the monitor is believed over the raw text which may be ISO-8859-2
So that made us think, that our config CHARSET setting (and therefore the NO_BINARY_CHECK) were not working.
After reading this article in the wiki:
http://www.splunk.com/wiki/Where_do_I_configure_my_Splunk_settings%3F
we moved the props.conf settings to our light forwarders
The ISO-8859-2 warnings disappeared and one log file of that type got indexed, but we have several such files. Some were missing.
I ended up deleting the fishbucket index on the light forwarder and all files are indexed properly now.
So I'm guessing that something in the fishbucket prevented those files from being indexed.
After reading this (old) blog post from Andrea Longo:
http://blogs.splunk.com/2008/08/14/what-is-this-fishbucket-thing/
I was hoping that I could search the _fishbucket index (on the light forwarder) and remove entries for the files that are not being indexed if I have a similar case in the future.
My first question is:
Is this a doable approach or have I missunderstood the problem/Is there a better way to resolve such issues?
My second question is
The fishbucket index on all our instances exists, but it is empty (viewing indexes from the Splunk Manager). How do I enable it on the indexer and is it possible to enable it and make it searchable on a SplunkLightForwarder somehow?
Thank you for helping me.
Edit--
Enabling the following debug settings $SPLUNK_HOME/etc/log.cfg helps showing whether new data from a file is detected by splunk
category.FileInputTracker=DEBUG
category.selectProcessor=DEBUG
category.TailingProcessor=DEBUG
This is documented in:
http://www.splunk.com/wiki/Community:Troubleshooting_Monitor_Inputs
... View more