Deployment Architecture

File growth rate must be higher than indexing or forwarding rate.

splunkranger
Path Finder

03-19-2014 18:47:22.017 -0500 INFO BatchReader - Will retry path="\servername1\dnsdebug\dns.log" after deferring for 10000ms, initCRC changed after being queued (before=0xc3e428fff4486a47, after=0x279979c853e7447a). File growth rate must be higher than indexing or forwarding rate.

I am not able to find any information on this error, can anyone comment?

Tags (2)

dmaislin_splunk
Splunk Employee
Splunk Employee

Might this help?

In $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/local/limits.conf

[thruput]
maxKBps = 0 # means unlimited

dmaislin_splunk
Splunk Employee
Splunk Employee

Please check my answer and accept it. Thanks!

0 Karma

u788332
New Member

I'm seeing same issue but for only one sourcetype , where ~ 10 sourcetypes are specified. I get this for the one sourcetype "File growth rate must be higher than indexing or forwarding rate" , limits.conf is set to 512kbps and is working well for other sourcetypes. The result is a large gaps in indexing for this specific log. However this log is very active many lines per second. Can a bottleneck occur only for high written logs? you would think it would happen on all sourcetypes at the same point in time.. thanks

0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee

You can get creative and run another forwarder on different ports on the same system when there is a scenario as you described. It helps eliminate that bottleneck where you can dedicate that forwarder to just that one heavy sourcetype and adjust that specific limits.conf for that.

0 Karma

splunkranger
Path Finder

That did solve the issue, just surprised there was no more information available on this error.
Thank you!

0 Karma

somesoni2
Revered Legend

It seems before Splunk can read the file, its getting changed, especially first few character which Splunk uses for CRC check. Does the program generates the log file overwrites the log file?

0 Karma
Get Updates on the Splunk Community!

This Week's Community Digest - Splunk Community Happenings [9.26.22]

Get the latest news and updates from the Splunk Community here! Upcoming User Group Events! 👏 Check ...

BSides Splunk 2022 - The Call for Papers is now Open!

TLDR; Main Site: https://bsidessplunk.com CFP Site: https://bsidessplunk.com/cfp CFP Opens: December 15th, ...

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...