Getting Data In

Why can't Splunk index some files in a directory?

edrivera3
Builder

Hi

I am trying to index a file from different subdirectory but Splunk is not indexing some of those files for some weird reason. All subdirectories contain different files but I am just interested in a file with an specific extension (.tir) so I am using a whilelist (.tir$). Splunk indexed almost all files but there are some files that Splunk just didn't index. Do you know a reason why is this happening?

I examined the files and they are normal (Same type data, same type format, and same extension).

0 Karma
1 Solution

edrivera3
Builder

I found a solution to my own problem. I need to add:
crcSalt =
initCrcLength = 2000

View solution in original post

edrivera3
Builder

I found a solution to my own problem. I need to add:
crcSalt =
initCrcLength = 2000

nawazns5038
Builder

What exactly does the setting do ?

0 Karma

bandit
Motivator

Essentially how many bytes in Splunk will check at the beginning of a file to try to uniquely identify it.

From: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf
initCrcLength =
* How much of a file, in bytes, that the input reads before trying to
identify whether it is a file that has already been seen. You might want to
adjust this if you have many files with common headers (comment headers,
long CSV headers, etc) and recurring filenames.
* Cannot be less than 256 or more than 1048576.
* CAUTION: Improper use of this setting will cause data to be re-indexed. You
might want to consult with Splunk Support before adjusting this value - the
default is fine for most installations.
* Default: 256 (bytes).

0 Karma

harsmarvania57
Ultra Champion

Please use below code in monitor stanza

whiltelist = (*.tir)$
recursive = true
0 Karma

edrivera3
Builder

I am already indexing recursively but Splunk is not indexing for some subdirectories.

0 Karma

harsmarvania57
Ultra Champion

Can you please check on forwarder with below command whether all files with (.tlr) is showing or not in command output?

 $SPLUNK_HOME/bin/splunk list monitor
0 Karma

edrivera3
Builder

I'm working with Windows OS.

0 Karma

edrivera3
Builder

I was examining the files and I noticed that there are very similar, almost the same. Is it possible that Splunk believed I am duplicating a file? If so, how could I change it so Splunk would index the file?

0 Karma

edrivera3
Builder

I didn't solve the problem. I checked the wrong index...

Anyway I tried to use CHECK_METHOD AND crcSalt but they didn't change anything.

0 Karma

edrivera3
Builder

But I found many errors in index=_internal which is weird because I tried using crcSalt = and the problem wasn't resolve. Looks like I have added these files previously which is wrong because I just created the index.

ERROR TailingProcessor - File will not be read, seekptr checksum did not match (file=C:\blah\blah\30343.tir). Last time we saw this initcrc, filename was different. You may wish to use a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

0 Karma

edrivera3
Builder

Well in Splunk>Data Input> Files & Directory appears that there 4835 files which contain the .tir extension and they are supposed to be indexed.

0 Karma

harsmarvania57
Ultra Champion

Then use below command

$SPLUNK_HOME/bin/splunk.exe list monitor

For example:

C:\Program Files\SplunkUniversalForwarder\bin>splunk.exe list monitor
0 Karma

edrivera3
Builder

I checked the current size of the index (36MB) and the event count (1,854) which look normal.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...