topic Re: Why can't Splunk index some files in a directory? in Getting Data In

Why can't Splunk index some files in a directory?

edrivera3 — Thu, 09 Apr 2015 15:38:01 GMT

I am trying to index a file from different subdirectory but Splunk is not indexing some of those files for some weird reason. All subdirectories contain different files but I am just interested in a file with an specific extension (.tir) so I am using a whilelist (.tir$). Splunk indexed almost all files but there are some files that Splunk just didn't index. Do you know a reason why is this happening?

I examined the files and they are normal (Same type data, same type format, and same extension).

Re: Why can't Splunk index some files in a directory?

edrivera3 — Thu, 09 Apr 2015 15:43:45 GMT

I checked the current size of the index (36MB) and the event count (1,854) which look normal.

Re: Why can't Splunk index some files in a directory?

harsmarvania57 — Thu, 09 Apr 2015 15:46:17 GMT

Please use below code in monitor stanza

whiltelist = (*.tir)$
recursive = true

Re: Why can't Splunk index some files in a directory?

edrivera3 — Thu, 09 Apr 2015 15:53:03 GMT

I am already indexing recursively but Splunk is not indexing for some subdirectories.

Re: Why can't Splunk index some files in a directory?

harsmarvania57 — Thu, 09 Apr 2015 16:08:49 GMT

Can you please check on forwarder with below command whether all files with (.tlr) is showing or not in command output?

 $SPLUNK_HOME/bin/splunk list monitor

Re: Why can't Splunk index some files in a directory?

edrivera3 — Thu, 09 Apr 2015 16:19:00 GMT

I'm working with Windows OS.

Re: Why can't Splunk index some files in a directory?

harsmarvania57 — Thu, 09 Apr 2015 16:22:40 GMT

Then use below command

$SPLUNK_HOME/bin/splunk.exe list monitor

For example:

C:\Program Files\SplunkUniversalForwarder\bin>splunk.exe list monitor

Re: Why can't Splunk index some files in a directory?

edrivera3 — Thu, 09 Apr 2015 16:23:38 GMT

Well in Splunk>Data Input> Files & Directory appears that there 4835 files which contain the .tir extension and they are supposed to be indexed.

Re: Why can't Splunk index some files in a directory?

edrivera3 — Thu, 09 Apr 2015 16:39:59 GMT

I was examining the files and I noticed that there are very similar, almost the same. Is it possible that Splunk believed I am duplicating a file? If so, how could I change it so Splunk would index the file?

Re: Why can't Splunk index some files in a directory?

edrivera3 — Thu, 09 Apr 2015 18:28:18 GMT

I didn't solve the problem. I checked the wrong index...

Anyway I tried to use CHECK_METHOD AND crcSalt but they didn't change anything.

Re: Why can't Splunk index some files in a directory?

edrivera3 — Thu, 09 Apr 2015 20:27:54 GMT

But I found many errors in index=_internal which is weird because I tried using crcSalt = and the problem wasn't resolve. Looks like I have added these files previously which is wrong because I just created the index.

ERROR TailingProcessor - File will not be read, seekptr checksum did not match (file=C:\blah\blah\30343.tir). Last time we saw this initcrc, filename was different. You may wish to use a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

Re: Why can't Splunk index some files in a directory?

edrivera3 — Fri, 10 Apr 2015 15:45:39 GMT

I found a solution to my own problem. I need to add:
crcSalt =
initCrcLength = 2000

Re: Why can't Splunk index some files in a directory?

nawazns5038 — Tue, 13 Nov 2018 22:46:16 GMT

What exactly does the setting do ?

Re: Why can't Splunk index some files in a directory?

bandit — Wed, 14 Nov 2018 02:59:09 GMT

Essentially how many bytes in Splunk will check at the beginning of a file to try to uniquely identify it.

From: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf
initCrcLength =
* How much of a file, in bytes, that the input reads before trying to
identify whether it is a file that has already been seen. You might want to
adjust this if you have many files with common headers (comment headers,
long CSV headers, etc) and recurring filenames.
* Cannot be less than 256 or more than 1048576.
* CAUTION: Improper use of this setting will cause data to be re-indexed. You
might want to consult with Splunk Support before adjusting this value - the
default is fine for most installations.
* Default: 256 (bytes).