Why did universal forwarder stop forwarding data?

[Eshwar]

Hi Community,

We have installed Universal forwarder on windows 2019 server and were able to get the data into Splunk. Since yesterday, the Universal forwarder stopped forwarding data to the indexer. No change in Network and configuration. We have identified below error while troubleshooting the issue. 

ERROR TcpOutputFd [4124 TcpOutEloop] - Connection to host=xx.xx.xx.xx:9997 failed
06-13-2023 00:11:28.769 -0700 WARN AutoLoadBalancedConnectionStrategy [4124 TcpOutEloop] - Applying quarantine to ip=xx.xx.xx.xx port=9997 connid=0 _numberOfFailures=2
06-13-2023 00:11:47.944 -0700 WARN TcpOutputProc [7272 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=xx.xx.xx.xx inside output group default-autolb-group from host_src=hostname1 has been blocked for blocked_seconds=1300. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
06-13-2023 00:12:02.123 -0700 INFO HttpPubSubConnection [4976 HttpClientPollingThread_D1664EB5-096A-4F59-8E50-70D7FB5CDD49] - Running phone uri=/services/broker/phonehome/connection_xx.xx.xx.xx_8089_xx.xx.xx.xx_hostname1_D1664EB5-096A-4F59-8E50-70D7FB5CDD49
06-13-2023 00:13:02.167 -0700 INFO HttpPubSubConnection [4976 HttpClientPollingThread_D1664EB5-096A-4F59-8E50-70D7FB5CDD49] - Running phone uri=/services/broker/phonehome/connection_xx.xx.xx.xx_8089_xx.xx.xx.xx_hostname1_D1664EB5-096A-4F59-8E50-70D7FB5CDD49
06-13-2023 00:13:28.222 -0700 WARN TcpOutputProc [7272 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=xx.xx.xx.xx inside output group default-autolb-group from host_src=hostname1 has been blocked for blocked_seconds=1400. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
06-13-2023 00:14:02.186 -0700 INFO HttpPubSubConnection [4976 HttpClientPollingThread_D1664EB5-096A-4F59-8E50-70D7FB5CDD49] - Running phone uri=/services/broker/phonehome/connection_xx.xx.xx.xx_8089_xx.xx.xx.xx_hostname1_D1664EB5-096A-4F59-8E50-70D7FB5CDD49
06-13-2023 00:15:02.197 -0700 INFO HttpPubSubConnection [4976 HttpClientPollingThread_D1664EB5-096A-4F59-8E50-70D7FB5CDD49] - Running phone uri=/services/broker/phonehome/connection_xx.xx.xx.xx_8089_xx.xx.xx.xx_hostname1_D1664EB5-096A-4F59-8E50-70D7FB5CDD49
06-13-2023 00:15:08.542 -0700 WARN TcpOutputProc [7272 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=xx.xx.xx.xx inside output group default-autolb-group from host_src=hostname1 has been blocked for blocked_seconds=1500. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

Please help us to resolve the issue.

[richgalloway]

The first log message is key: the UF lost the connection to the indexer.  Verify the indexer is still running and using port 9997.  Confirm the UF is allowed to connect to that address and port.

[Eshwar]

Hi @richgalloway,

Yes, Indexer is running and other universal forwarders sending data to indexer. while doing telnet on port 9997 from universal forwarder then it refusing the connection. We have disabled firewall in both servers.

[richgalloway]

Have you tried restarting the UF?

[isoutamo]

When you said "refused connection" what you are actually meaning? Did it drop the connection, refused it or was it splunkd which are refused it?

What you are founding on splunkd.log on indexer side?