Installation

UF to HF - no active forwards

dabroma5
Explorer

Hi,

I am trying to configure Universal Forwarder and Heavy forwarder.

In UF  I see:

Active forwards:
None
Configured but inactive forwards:
A.B.C.D:9997

splunkd.log:

07-23-2021 11:45:00.807 +0000 WARN AutoLoadBalancedConnectionStrategy [42092 TcpOutEloop] - Applying quarantine to ip=A.B.C.D port=9997 _numberOfFailures=2
07-23-2021 11:45:42.188 +0000 WARN TcpOutputProc [42091 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=A.B.C.D inside output group default-autolb-group from host_src=UF_name has been blocked for blocked_seconds=3000. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
07-23-2021 11:47:22.196 +0000 WARN TcpOutputProc [42091 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=A.B.C.D inside output group default-autolb-group from host_src=UF_name has been blocked for blocked_seconds=3100. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
07-23-2021 11:49:02.204 +0000 WARN TcpOutputProc [42091 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=A.B.C.D inside output group default-autolb-group from host_src=UF_name has been blocked for blocked_seconds=3200. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
07-23-2021 11:50:29.730 +0000 INFO AutoLoadBalancedConnectionStrategy [42092 TcpOutEloop] - Removing quarantine from idx=A.B.C.D:9997
07-23-2021 11:50:29.732 +0000 ERROR TcpOutputFd [42092 TcpOutEloop] - Read error. Connection reset by peer
07-23-2021 11:50:29.734 +0000 ERROR TcpOutputFd [42092 TcpOutEloop] - Read error. Connection reset by peer
07-23-2021 11:50:29.734 +0000 WARN AutoLoadBalancedConnectionStrategy [42092 TcpOutEloop] - Applying quarantine to ip=A.B.C.D port=9997 _numberOfFailures=2

 

tcpdump also showed me reset from HF side. 

 

I have communication between UF and HF - all necessary ports are open. 

[root@UF_name ~]# nc -z -v A.B.C.D 9997
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to A.B.C.D:9997.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
[root@UF_name ~]# nc -z -v A.B.C.D 8000
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to A.B.C.D6:8000.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
[root@UF_name ~]# nc -z -v A.B.C.D 8089
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to A.B.C.D:8089.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.

 

How to solve this problem? Any tips?

 

Labels (4)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Check the logs on the HF side to see if it explains why the HF is dropping the connections.

If you're using SSL, verify the certificates are correct on both sides.

---
If this reply helps you, Karma would be appreciated.
0 Karma

dabroma5
Explorer

Hi

Do you mean splunkd.log? 

SSL is only used towards cloud

0 Karma
Get Updates on the Splunk Community!

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...

Announcing General Availability of Splunk Incident Intelligence!

Digital transformation is real! Across industries, companies big and small are going through rapid digital ...