SplunkForwarder flooding splunkd.log with reconnect errors

[yachtbum]

After upgrading to splunkforwarder 9.4.5 we noticed that SplunkForwarder.service gets panic when loosing connection to its endpoint.

splunkd.log gets flooded with these messages:
 
10-10-2025 14:22:48.336 +0100 ERROR TcpOutputFd [234730 TcpOutEloop] - Connection to host=xx.xx.xx.xx:xxxx failed
10-10-2025 14:22:48.337 +0100 WARN  TcpOutputFd [234730 TcpOutEloop] - Connect to xx.xx.xx.xx:xxxx failed. Connection refused

When doing a count in the logfile it looks like the forwarder tries to reconnect about 5000 times every second.

Anyone else noticed this behaviour?
Is it possible to configure the reconnect retry timeout to make the forwarder more relaxed ? 

[shashankD]

Increase maxkbps value to 0 (unlimited) and maxThruput to 5mb, this will resolve the issue

[VatsalJagani]

@yachtbum - This is true in some environments. I notice so many of these warnings. Not sure exact root cause.

Although you cannot currently configure the reconnect retry timeout in Splunk. 

But you can configure the normal connection timeout setting in outputs.conf on the forwarder if that is what you need for your environment.

VatsalJagani_0-1762608518763.png

I hope this helps!!! Kindly upvote if it does!!