Deployment Architecture

File growth rate must be higher than indexing or forwarding rate.

splunkranger
Path Finder

03-19-2014 18:47:22.017 -0500 INFO BatchReader - Will retry path="\servername1\dnsdebug\dns.log" after deferring for 10000ms, initCRC changed after being queued (before=0xc3e428fff4486a47, after=0x279979c853e7447a). File growth rate must be higher than indexing or forwarding rate.

I am not able to find any information on this error, can anyone comment?

Tags (2)

dmaislin_splunk
Splunk Employee
Splunk Employee

Might this help?

In $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/local/limits.conf

[thruput]
maxKBps = 0 # means unlimited

dmaislin_splunk
Splunk Employee
Splunk Employee

Please check my answer and accept it. Thanks!

0 Karma

u788332
New Member

I'm seeing same issue but for only one sourcetype , where ~ 10 sourcetypes are specified. I get this for the one sourcetype "File growth rate must be higher than indexing or forwarding rate" , limits.conf is set to 512kbps and is working well for other sourcetypes. The result is a large gaps in indexing for this specific log. However this log is very active many lines per second. Can a bottleneck occur only for high written logs? you would think it would happen on all sourcetypes at the same point in time.. thanks

0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee

You can get creative and run another forwarder on different ports on the same system when there is a scenario as you described. It helps eliminate that bottleneck where you can dedicate that forwarder to just that one heavy sourcetype and adjust that specific limits.conf for that.

0 Karma

splunkranger
Path Finder

That did solve the issue, just surprised there was no more information available on this error.
Thank you!

0 Karma

somesoni2
Revered Legend

It seems before Splunk can read the file, its getting changed, especially first few character which Splunk uses for CRC check. Does the program generates the log file overwrites the log file?

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

Tech Talk Recap | Mastering Threat Hunting

Mastering Threat HuntingDive into the world of threat hunting, exploring the key differences between ...

Observability for AI Applications: Troubleshooting Latency

If you’re working with proprietary company data, you’re probably going to have a locally hosted LLM or many ...

Splunk AI Assistant for SPL vs. ChatGPT: Which One is Better?

In the age of AI, every tool promises to make our lives easier. From summarizing content to writing code, ...