Getting Data In

How to monitor a file input with the same filename?


I have an input, which is a CSV file. I want to use this as a batch input. The file is generated every day, with the same name.

Most of the time, the beginning of the file is also the same. That's why I think the only method I can achieve to have this ingested every day is to use the CHECK_METHOD = modtime in props.conf
After setting it, I've checked my splunkd.log, I can see that because of the new modtime, Splunk started to read the file, but now I'm getting this messages:

TailReader - Will retry path="" after deferring for 10000ms, initCRC changed after being queued (before=0xd71b3bcce79fb3d8, after=0xb5a12ee23beafb6a).  File growth rate must be higher than indexing or forwarding rate.

I've tried restarting my forwarder, deleting the fishbucket, but I'm still getting this error. (The file wasn't modified during this time)

0 Karma

Splunk Employee
Splunk Employee

If your csv file has header, you will need to increase the size of the CRC check splunk is doing to identify the file. This can be done using the parameter initCRCLength as indicated in the inputs.conf.spec file:

initCrcLength =
* This setting adjusts how much of a file Splunk reads before trying to
identify whether it is a file that has already been seen. You may want to
adjust this if you have many files with common headers (comment headers, long
CSV headers, etc) and recurring filenames.
* CAUTION: Improper use of this setting will cause data to be reindexed. You
may wish to consult with Splunk Support before adjusting this value - the
default is fine for most installations.
* Defaults to 256 (bytes).
* Must be in the range 256-1048576.

0 Karma