How to troubleshoot why a file in monitored folder is not being indexed in Splunk Light Free 6.3.0?

[gbeddow]

Hi,

I’ve been using Splunk Light Free Version 6.3.0 for about a month on Mac OS X, and it’s been working well, monitoring 16 modestly-sized log files updated once per day, taken from 3 different websites.

Here’s the relevant part of inputs.conf:

[monitor:///Users/user/backup-log]
disabled = false
host = user.com
blacklist = \/\.[^\/]*$
sourcetype=access_combined

Recently I noticed a newly-added log file (#17) in a monitored folder wasn’t being indexed. At first thinking there may be a limit on the number of sources in the free version, I removed a couple of older log files from the monitored folder, then cleaned out all the index data and reindexed everything:

$ /Applications/Splunk/bin/splunk stop
$ /Applications/Splunk/bin/splunk clean eventdata
$ /Applications/Splunk/bin/splunk start

But it simply indexed 14 files instead of 16, still omitting the newly-added log file.

So then I tried renaming the newly-added log file and reindexing everything. Interestingly, the renamed file got indexed, but some other file was no longer indexed (and it was being indexed before).

Any ideas out there on how to fix this?

Thanks,
Greg

[MuS · Accepted solution]

Hi gbeddow,

check splunkd.log or index=_internal for any message related to the missing file. Do some reading here http://docs.splunk.com/Documentation/Splunk/6.3.0/Data/HowLogFileRotationIsHandled and learn how Splunk uses the file header to do CRC; maybe you have to adjust initCrcLength in inputs.conf http://docs.splunk.com/Documentation/Splunk/6.3.0/admin/Inputsconf

Also

• Check the file permission of this missing field, the user running Splunk should be able to read it
• Make sure the blacklist is not matching the missing file
• Have a look at this superb blog http://blogs.splunk.com/2011/01/02/did-i-miss-christmas-2/ to learn more about the TailingProcess

Hope this helps ...

cheers, MuS

[gbeddow]

Hi MuS,

I looked at splunkd.log and found this from when Splunk first saw the newly-added log file in question:

11-01-2015 06:56:44.400 -0800 ERROR TailReader - File will not be read, is too small to match seekptr checksum (file=<some-file-name>).  Last time we saw this initcrc, filename was different.  You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source.  Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

Then I looked at these:

http://docs.splunk.com/Documentation/Splunk/6.3.0/Data/HowLogFileRotationIsHandled
http://docs.splunk.com/Documentation/Splunk/6.3.0/Data/Monitorfilesanddirectorieswithinputs.conf

So I added this to each stanza in inputs.conf:

crcSalt = <SOURCE>

then reindexed everything and that took care of the problem.

Thanks for your help!

--Greg

[BansodeSantosh]

crcSalt =
This worked for me....Great!!

[MuS]

Watch out when using crcSalt = this can give you double indexed events! If you encounter double indexed events, change the initCrcLen and discard crcSalt = 😉