topic Re: File growth rate must be higher than indexing or forwarding rate in Getting Data In

File growth rate must be higher than indexing or forwarding rate

Rakzskull — Fri, 17 Jan 2025 13:47:01 GMT

Hi guys,

I am currently encountering an error that is affecting performance, resulting in delays with the file processing. If anyone has insights or solutions to address this issue.

01-17-2025 04:33:12.580 -0600 INFO TailReader [1853894 batchreader0] - Will retry path="/apps2.log" after deferring for 10000ms, initCRC changed after being queued (before=0x47710a7c475501b6, after=0x23c7e0f63f123bf1). File growth rate must be higher than indexing or forwarding rate.

01-17-2025 04:20:24.672 -0600 WARN TailReader [1544431 tailreader0] - Enqueuing a very large file=/apps2.log in the batch reader, with bytes_to_read=292732393, reading of other large files could be delayed

I would greatly appreciate your assistance.

Thank you.

Re: File growth rate must be higher than indexing or forwarding rate

_gkollias — Fri, 17 Jan 2025 14:28:21 GMT

Hi Rakzskull, what is the maxKBps set to in limits.conf on the Forwarder where you see this message? By default it is 256. If possible, you could try increasing thruput. Also, this doc may be helpful:
https://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Troubleshootingeventsindexingdela...

It might be worth investigating why the file is so large, whether the data being logged is excessive, and if there are opportunities to optimize its size, rotation frequency, etc. Hope this helps!

Re: File growth rate must be higher than indexing or forwarding rate

Rakzskull — Fri, 17 Jan 2025 16:24:42 GMT

@_gkollias It already set it to "0" (unlimited) , Is there anything I should update?

Re: File growth rate must be higher than indexing or forwarding rate

isoutamo — Fri, 17 Jan 2025 19:51:49 GMT

Another thing which may help you is adding parallelIngestionPipelines > 1 in server.conf. This is not helping with individual files, but if there are many files then it could help.

Re: File growth rate must be higher than indexing or forwarding rate

Rakzskull — Sat, 18 Jan 2025 02:08:30 GMT

I have this conf in server.conf parallelIngestionPipelines = 2 still getting same issue

Re: File growth rate must be higher than indexing or forwarding rate

PickleRick — Sat, 18 Jan 2025 11:53:18 GMT

It seems you have big and frequently changing files and your forwarder can't keep up with reading them. That's the problem. But the root cause can be really anywhere, depending on your infrastructure. I assume you're pushing it to the Cloud but maybe your network connection can't handle the traffic. Or your Cloud indexers can't handle the amount of data. Or you're reading the files in an inefficient way (for example - from a networked filesystem)...

There can be many reasons.

Re: File growth rate must be higher than indexing or forwarding rate

isoutamo — Sun, 19 Jan 2025 11:13:46 GMT

Here is one conf presentation which could help you https://conf.splunk.com/files/2019/slides/FN1570.pdf?_gl=1*1l6tz7s*_gcl_aw*R0NMLjE3MzA4NDM5NTUuRUFJYUlRb2JDaE1JcFpEUWtaakdpUU1WbEZPUkJSMENNZ2YzRUFBWUFpQUFFZ0pVbFBEX0J3RQ..*_gcl_au*MTIyNzAyMTUzNC4xNzMxNTA5NTYz*FPAU*MTIyNzAyMTUzNC4xNzMxNTA5NTYz*_ga*MTM3Njk1MTUzMy4xNzIzMzcyODc0*_ga_5EPM2P39FV*MTczNzI4NDk0Ny4yMjYuMS4xNzM3Mjg1MTE2LjAuMC4xMTE5MjkyNjg2*_fplc*eW9LSGpxJTJGQjlXVm8zVzk5UTZwNHB5ZkEwcW96TjdlaURYUzF1RkY3d0tORVlQaXQ2N2pSaU9aUzNqQXhvVUQ2SkpJcFB6JTJGSHZiZlJyOXE5dFJHUVlvMmRiZmM4a0FYTzlvRGVqUkgyV3hDOEthY3Y1Y1c5SWJEWUNMaVBadyUzRCUzRA..
But as @PickleRick said there could be many reasons behind that issue.

Re: File growth rate must be higher than indexing or forwarding rate

Rakzskull — Tue, 21 Jan 2025 15:01:04 GMT

After increasing the pipeline to 4, I have observed some improvements.

Thanks

Re: File growth rate must be higher than indexing or forwarding rate

isoutamo — Tue, 21 Jan 2025 16:02:30 GMT

Nice, but ensure that you cannot add too many pipelines. Usually it’s less than you have cores/cpus on your box. And this is valid only for forwarders.

Re: File growth rate must be higher than indexing or forwarding rate

Rakzskull — Tue, 25 Feb 2025 07:47:04 GMT

Here is my final configuration to resolve this issue. I hope it assists anyone experiencing a similar problem in finding a solution.

UF --> $SPLUNK/etc/system/local/server.conf

[queue=parsingQueue]
maxSize = 20MB

[general]
parallelIngestionPipelines = 4

Re: File growth rate must be higher than indexing or forwarding rate

isoutamo — Fri, 28 Feb 2025 14:47:28 GMT

One proposal. You should always create a splunk TA/SA or other app which contains those fixes and can be installed easily for other nodes too.
Personally I avoid as much as possible to put anything into .../etc/system/local/. You cannot easily change those values after those are there Of course if you have any SM tool which can edit / install those then it's different story.

Re: File growth rate must be higher than indexing or forwarding rate

PickleRick — Fri, 28 Feb 2025 18:45:58 GMT

+1 on thay. Generally you should avoid putting anything in system/local.

Those settings have the highest priority and cannot be overriden by apps (with an exception for indexer cluster apps pushed from manager into peer-apps so you cannot manage those settings easily without physically touching the server.