Deployment Architecture

File growth rate must be higher than indexing or forwarding rate.

Path Finder

03-19-2014 18:47:22.017 -0500 INFO BatchReader - Will retry path="\servername1\dnsdebug\dns.log" after deferring for 10000ms, initCRC changed after being queued (before=0xc3e428fff4486a47, after=0x279979c853e7447a). File growth rate must be higher than indexing or forwarding rate.

I am not able to find any information on this error, can anyone comment?

Tags (2)

Splunk Employee
Splunk Employee

Might this help?

In $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/local/limits.conf

[thruput]
maxKBps = 0 # means unlimited

Splunk Employee
Splunk Employee

Please check my answer and accept it. Thanks!

0 Karma

New Member

I'm seeing same issue but for only one sourcetype , where ~ 10 sourcetypes are specified. I get this for the one sourcetype "File growth rate must be higher than indexing or forwarding rate" , limits.conf is set to 512kbps and is working well for other sourcetypes. The result is a large gaps in indexing for this specific log. However this log is very active many lines per second. Can a bottleneck occur only for high written logs? you would think it would happen on all sourcetypes at the same point in time.. thanks

0 Karma

Splunk Employee
Splunk Employee

You can get creative and run another forwarder on different ports on the same system when there is a scenario as you described. It helps eliminate that bottleneck where you can dedicate that forwarder to just that one heavy sourcetype and adjust that specific limits.conf for that.

0 Karma

Path Finder

That did solve the issue, just surprised there was no more information available on this error.
Thank you!

0 Karma

SplunkTrust
SplunkTrust

It seems before Splunk can read the file, its getting changed, especially first few character which Splunk uses for CRC check. Does the program generates the log file overwrites the log file?

0 Karma