UF to HF - no active forwards

[dabroma5]

Hi,

I am trying to configure Universal Forwarder and Heavy forwarder.

In UF  I see:

Active forwards:
None
Configured but inactive forwards:
A.B.C.D:9997

splunkd.log:

07-23-2021 11:45:00.807 +0000 WARN AutoLoadBalancedConnectionStrategy [42092 TcpOutEloop] - Applying quarantine to ip=A.B.C.D port=9997 _numberOfFailures=2
07-23-2021 11:45:42.188 +0000 WARN TcpOutputProc [42091 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=A.B.C.D inside output group default-autolb-group from host_src=UF_name has been blocked for blocked_seconds=3000. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
07-23-2021 11:47:22.196 +0000 WARN TcpOutputProc [42091 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=A.B.C.D inside output group default-autolb-group from host_src=UF_name has been blocked for blocked_seconds=3100. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
07-23-2021 11:49:02.204 +0000 WARN TcpOutputProc [42091 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=A.B.C.D inside output group default-autolb-group from host_src=UF_name has been blocked for blocked_seconds=3200. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
07-23-2021 11:50:29.730 +0000 INFO AutoLoadBalancedConnectionStrategy [42092 TcpOutEloop] - Removing quarantine from idx=A.B.C.D:9997
07-23-2021 11:50:29.732 +0000 ERROR TcpOutputFd [42092 TcpOutEloop] - Read error. Connection reset by peer
07-23-2021 11:50:29.734 +0000 ERROR TcpOutputFd [42092 TcpOutEloop] - Read error. Connection reset by peer
07-23-2021 11:50:29.734 +0000 WARN AutoLoadBalancedConnectionStrategy [42092 TcpOutEloop] - Applying quarantine to ip=A.B.C.D port=9997 _numberOfFailures=2

tcpdump also showed me reset from HF side. 

I have communication between UF and HF - all necessary ports are open. 

[root@UF_name ~]# nc -z -v A.B.C.D 9997
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to A.B.C.D:9997.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
[root@UF_name ~]# nc -z -v A.B.C.D 8000
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to A.B.C.D6:8000.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
[root@UF_name ~]# nc -z -v A.B.C.D 8089
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to A.B.C.D:8089.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.

How to solve this problem? Any tips?

[richgalloway]

Check the logs on the HF side to see if it explains why the HF is dropping the connections.

If you're using SSL, verify the certificates are correct on both sides.

[dabroma5]

Hi

Do you mean splunkd.log? 

SSL is only used towards cloud