topic Re: How to troubleshoot why a file in monitored folder is not being indexed in Splunk Light Free 6.3.0? in Getting Data In

How to troubleshoot why a file in monitored folder is not being indexed in Splunk Light Free 6.3.0?

gbeddow — Tue, 03 Nov 2015 19:13:52 GMT

Hi,

I’ve been using Splunk Light Free Version 6.3.0 for about a month on Mac OS X, and it’s been working well, monitoring 16 modestly-sized log files updated once per day, taken from 3 different websites.

Here’s the relevant part of inputs.conf:

[monitor:///Users/user/backup-log]
disabled = false
host = user.com
blacklist = \/\.[^\/]*$
sourcetype=access_combined

Recently I noticed a newly-added log file (#17) in a monitored folder wasn’t being indexed. At first thinking there may be a limit on the number of sources in the free version, I removed a couple of older log files from the monitored folder, then cleaned out all the index data and reindexed everything:

$ /Applications/Splunk/bin/splunk stop
$ /Applications/Splunk/bin/splunk clean eventdata
$ /Applications/Splunk/bin/splunk start

But it simply indexed 14 files instead of 16, still omitting the newly-added log file.

So then I tried renaming the newly-added log file and reindexing everything. Interestingly, the renamed file got indexed, but some other file was no longer indexed (and it was being indexed before).

Any ideas out there on how to fix this?

Thanks,
Greg

Re: How to troubleshoot why a file in monitored folder is not being indexed in Splunk Light Free 6.3.0?

MuS — Tue, 03 Nov 2015 19:39:19 GMT

Hi gbeddow,

check splunkd.log or index=_internal for any message related to the missing file. Do some reading here http://docs.splunk.com/Documentation/Splunk/6.3.0/Data/HowLogFileRotationIsHandled and learn how Splunk uses the file header to do CRC; maybe you have to adjust initCrcLength in inputs.conf http://docs.splunk.com/Documentation/Splunk/6.3.0/admin/Inputsconf

Also

Check the file permission of this missing field, the user running Splunk should be able to read it
Make sure the blacklist is not matching the missing file
Have a look at this superb blog http://blogs.splunk.com/2011/01/02/did-i-miss-christmas-2/ to learn more about the TailingProcess

Hope this helps ...

cheers, MuS

Re: How to troubleshoot why a file in monitored folder is not being indexed in Splunk Light Free 6.3.0?

gbeddow — Tue, 03 Nov 2015 20:29:28 GMT

Hi MuS,

I looked at splunkd.log and found this from when Splunk first saw the newly-added log file in question:

11-01-2015 06:56:44.400 -0800 ERROR TailReader - File will not be read, is too small to match seekptr checksum (file=<some-file-name>).  Last time we saw this initcrc, filename was different.  You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source.  Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

Then I looked at these:

http://docs.splunk.com/Documentation/Splunk/6.3.0/Data/HowLogFileRotationIsHandled
http://docs.splunk.com/Documentation/Splunk/6.3.0/Data/Monitorfilesanddirectorieswithinputs.conf

So I added this to each stanza in inputs.conf:

crcSalt = <SOURCE>

then reindexed everything and that took care of the problem.

Thanks for your help!

--Greg

Re: How to troubleshoot why a file in monitored folder is not being indexed in Splunk Light Free 6.3.0?

MuS — Tue, 03 Nov 2015 20:33:01 GMT

Watch out when using crcSalt = this can give you double indexed events! If you encounter double indexed events, change the initCrcLen and discard crcSalt = 😉

Re: How to troubleshoot why a file in monitored folder is not being indexed in Splunk Light Free 6.3.0?

BansodeSantosh — Mon, 08 May 2017 09:55:37 GMT

crcSalt =
This worked for me....Great!!