Getting Data In

How come my forwarders appear frozen and are unable to send data to the indexers?

Splunk Employee
Splunk Employee

We have a lot of forwarders, of universal forwarders and heavy forwarders mostly, in version 7.0.x, which are configured to send data to 10s of indexers.

Intermittently, some forwarders fail to send data putting a lot of messages in the splunkd.log then it becomes fine after a restart.

Around the time of incident, one of 10s indexers had disk issue and had connectivity issue.

======[ Splunkd.log ]==
WARN TcpOutputFd - Connect to 11.22.33.44:9997 failed. Connection refused
[ snip ]
WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 10 seconds.
[ snip ]
WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 5790 seconds.
==============================

1 Solution

Splunk Employee
Splunk Employee

We have seen this issues from time to time and have been working on it to get to the bottom of issue.
This can be worked around by using forcedTimebasedAutoLB = true until the fix is found and released.

View solution in original post

Splunk Employee
Splunk Employee

We have seen this issues from time to time and have been working on it to get to the bottom of issue.
This can be worked around by using forcedTimebasedAutoLB = true until the fix is found and released.

View solution in original post

Loves-to-Learn Lots

in which path and file should I enter that value? @sylim_splunk 

0 Karma