Getting Data In

Why can't Splunk index some files in a directory?

edrivera3
Builder

Hi

I am trying to index a file from different subdirectory but Splunk is not indexing some of those files for some weird reason. All subdirectories contain different files but I am just interested in a file with an specific extension (.tir) so I am using a whilelist (.tir$). Splunk indexed almost all files but there are some files that Splunk just didn't index. Do you know a reason why is this happening?

I examined the files and they are normal (Same type data, same type format, and same extension).

0 Karma
1 Solution

edrivera3
Builder

I found a solution to my own problem. I need to add:
crcSalt =
initCrcLength = 2000

View solution in original post

edrivera3
Builder

I found a solution to my own problem. I need to add:
crcSalt =
initCrcLength = 2000

nawazns5038
Builder

What exactly does the setting do ?

0 Karma

bandit
Motivator

Essentially how many bytes in Splunk will check at the beginning of a file to try to uniquely identify it.

From: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf
initCrcLength =
* How much of a file, in bytes, that the input reads before trying to
identify whether it is a file that has already been seen. You might want to
adjust this if you have many files with common headers (comment headers,
long CSV headers, etc) and recurring filenames.
* Cannot be less than 256 or more than 1048576.
* CAUTION: Improper use of this setting will cause data to be re-indexed. You
might want to consult with Splunk Support before adjusting this value - the
default is fine for most installations.
* Default: 256 (bytes).

0 Karma

harsmarvania57
Ultra Champion

Please use below code in monitor stanza

whiltelist = (*.tir)$
recursive = true
0 Karma

edrivera3
Builder

I am already indexing recursively but Splunk is not indexing for some subdirectories.

0 Karma

harsmarvania57
Ultra Champion

Can you please check on forwarder with below command whether all files with (.tlr) is showing or not in command output?

 $SPLUNK_HOME/bin/splunk list monitor
0 Karma

edrivera3
Builder

I'm working with Windows OS.

0 Karma

edrivera3
Builder

I was examining the files and I noticed that there are very similar, almost the same. Is it possible that Splunk believed I am duplicating a file? If so, how could I change it so Splunk would index the file?

0 Karma

edrivera3
Builder

I didn't solve the problem. I checked the wrong index...

Anyway I tried to use CHECK_METHOD AND crcSalt but they didn't change anything.

0 Karma

edrivera3
Builder

But I found many errors in index=_internal which is weird because I tried using crcSalt = and the problem wasn't resolve. Looks like I have added these files previously which is wrong because I just created the index.

ERROR TailingProcessor - File will not be read, seekptr checksum did not match (file=C:\blah\blah\30343.tir). Last time we saw this initcrc, filename was different. You may wish to use a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

0 Karma

edrivera3
Builder

Well in Splunk>Data Input> Files & Directory appears that there 4835 files which contain the .tir extension and they are supposed to be indexed.

0 Karma

harsmarvania57
Ultra Champion

Then use below command

$SPLUNK_HOME/bin/splunk.exe list monitor

For example:

C:\Program Files\SplunkUniversalForwarder\bin>splunk.exe list monitor
0 Karma

edrivera3
Builder

I checked the current size of the index (36MB) and the event count (1,854) which look normal.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk Observability Cloud – June 2025

What’s New in Splunk Observability Cloud – June 2025 We are excited to announce the latest enhancements to ...

Almost Too Eventful Assurance: Part 2

Work While You SleepBefore you can rely on any autonomous remediation measures, you need to close the loop ...

Leveraging Detections from the Splunk Threat Research Team & Cisco Talos

 Stay ahead of today’s evolving threats with the combined power of the Splunk Threat Research Team (STRT) and ...