Getting Data In

How to monitor a file input with the same filename?

szabados
Communicator

I have an input, which is a CSV file. I want to use this as a batch input. The file is generated every day, with the same name.

Most of the time, the beginning of the file is also the same. That's why I think the only method I can achieve to have this ingested every day is to use the CHECK_METHOD = modtime in props.conf
After setting it, I've checked my splunkd.log, I can see that because of the new modtime, Splunk started to read the file, but now I'm getting this messages:

TailReader - Will retry path="" after deferring for 10000ms, initCRC changed after being queued (before=0xd71b3bcce79fb3d8, after=0xb5a12ee23beafb6a).  File growth rate must be higher than indexing or forwarding rate.

I've tried restarting my forwarder, deleting the fishbucket, but I'm still getting this error. (The file wasn't modified during this time)

0 Karma

dgrubb_splunk
Splunk Employee
Splunk Employee

If your csv file has header, you will need to increase the size of the CRC check splunk is doing to identify the file. This can be done using the parameter initCRCLength as indicated in the inputs.conf.spec file:

initCrcLength =
* This setting adjusts how much of a file Splunk reads before trying to
identify whether it is a file that has already been seen. You may want to
adjust this if you have many files with common headers (comment headers, long
CSV headers, etc) and recurring filenames.
* CAUTION: Improper use of this setting will cause data to be reindexed. You
may wish to consult with Splunk Support before adjusting this value - the
default is fine for most installations.
* Defaults to 256 (bytes).
* Must be in the range 256-1048576.

0 Karma
Get Updates on the Splunk Community!

Cultivate Your Career Growth with Fresh Splunk Training

Growth doesn’t just happen—it’s nurtured. Like tending a garden, developing your Splunk skills takes the right ...

Introducing a Smarter Way to Discover Apps on Splunkbase

We’re excited to announce the launch of a foundational enhancement to Splunkbase: App Tiering.  Because we’ve ...

How to Send Splunk Observability Alerts to Webex teams in Minutes

As a Developer Evangelist at Splunk, my team and I are constantly tinkering with technology to explore its ...