Getting Data In

Why did universal forwarder stop forwarding data?

Eshwar
Engager

Hi Community,

We have installed Universal forwarder on windows 2019 server and were able to get the data into Splunk. Since yesterday, the Universal forwarder stopped forwarding data to the indexer. No change in Network and configuration. We have identified below error while troubleshooting the issue. 

ERROR TcpOutputFd [4124 TcpOutEloop] - Connection to host=xx.xx.xx.xx:9997 failed
06-13-2023 00:11:28.769 -0700 WARN AutoLoadBalancedConnectionStrategy [4124 TcpOutEloop] - Applying quarantine to ip=xx.xx.xx.xx port=9997 connid=0 _numberOfFailures=2
06-13-2023 00:11:47.944 -0700 WARN TcpOutputProc [7272 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=xx.xx.xx.xx inside output group default-autolb-group from host_src=hostname1 has been blocked for blocked_seconds=1300. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
06-13-2023 00:12:02.123 -0700 INFO HttpPubSubConnection [4976 HttpClientPollingThread_D1664EB5-096A-4F59-8E50-70D7FB5CDD49] - Running phone uri=/services/broker/phonehome/connection_xx.xx.xx.xx_8089_xx.xx.xx.xx_hostname1_D1664EB5-096A-4F59-8E50-70D7FB5CDD49
06-13-2023 00:13:02.167 -0700 INFO HttpPubSubConnection [4976 HttpClientPollingThread_D1664EB5-096A-4F59-8E50-70D7FB5CDD49] - Running phone uri=/services/broker/phonehome/connection_xx.xx.xx.xx_8089_xx.xx.xx.xx_hostname1_D1664EB5-096A-4F59-8E50-70D7FB5CDD49
06-13-2023 00:13:28.222 -0700 WARN TcpOutputProc [7272 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=xx.xx.xx.xx inside output group default-autolb-group from host_src=hostname1 has been blocked for blocked_seconds=1400. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
06-13-2023 00:14:02.186 -0700 INFO HttpPubSubConnection [4976 HttpClientPollingThread_D1664EB5-096A-4F59-8E50-70D7FB5CDD49] - Running phone uri=/services/broker/phonehome/connection_xx.xx.xx.xx_8089_xx.xx.xx.xx_hostname1_D1664EB5-096A-4F59-8E50-70D7FB5CDD49
06-13-2023 00:15:02.197 -0700 INFO HttpPubSubConnection [4976 HttpClientPollingThread_D1664EB5-096A-4F59-8E50-70D7FB5CDD49] - Running phone uri=/services/broker/phonehome/connection_xx.xx.xx.xx_8089_xx.xx.xx.xx_hostname1_D1664EB5-096A-4F59-8E50-70D7FB5CDD49
06-13-2023 00:15:08.542 -0700 WARN TcpOutputProc [7272 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=xx.xx.xx.xx inside output group default-autolb-group from host_src=hostname1 has been blocked for blocked_seconds=1500. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

Please help us to resolve the issue.

Labels (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

The first log message is key: the UF lost the connection to the indexer.  Verify the indexer is still running and using port 9997.  Confirm the UF is allowed to connect to that address and port.

---
If this reply helps you, Karma would be appreciated.
0 Karma

Eshwar
Engager

Hi @richgalloway,

Yes, Indexer is running and other universal forwarders sending data to indexer. while doing telnet on port 9997 from universal forwarder then it refusing the connection. We have disabled firewall in both servers.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Have you tried restarting the UF?

---
If this reply helps you, Karma would be appreciated.
0 Karma

isoutamo
SplunkTrust
SplunkTrust

When you said "refused connection" what you are actually meaning? Did it drop the connection, refused it or was it splunkd which are refused it?

What you are founding on splunkd.log on indexer side?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...