Getting Data In
Provide Splunk Cloud feedback in this confidential UX survey by June 17
for a chance to win a $200 Amazon gift card!

How come my forwarders appear frozen and are unable to send data to the indexers?

sylim_splunk
Splunk Employee
Splunk Employee

We have a lot of forwarders, of universal forwarders and heavy forwarders mostly, in version 7.0.x, which are configured to send data to 10s of indexers.

Intermittently, some forwarders fail to send data putting a lot of messages in the splunkd.log then it becomes fine after a restart.

Around the time of incident, one of 10s indexers had disk issue and had connectivity issue.

======[ Splunkd.log ]==
WARN TcpOutputFd - Connect to 11.22.33.44:9997 failed. Connection refused
[ snip ]
WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 10 seconds.
[ snip ]
WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 5790 seconds.
==============================

1 Solution

sylim_splunk
Splunk Employee
Splunk Employee

We have seen this issues from time to time and have been working on it to get to the bottom of issue.
This can be worked around by using forcedTimebasedAutoLB = true until the fix is found and released.

View solution in original post

sylim_splunk
Splunk Employee
Splunk Employee

We have seen this issues from time to time and have been working on it to get to the bottom of issue.
This can be worked around by using forcedTimebasedAutoLB = true until the fix is found and released.

View solution in original post

juancamiloll
Loves-to-Learn Lots

in which path and file should I enter that value? @sylim_splunk 

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!