Installation

UF to HF - no active forwards

dabroma5
Explorer

Hi,

I am trying to configure Universal Forwarder and Heavy forwarder.

In UF  I see:

Active forwards:
None
Configured but inactive forwards:
A.B.C.D:9997

splunkd.log:

07-23-2021 11:45:00.807 +0000 WARN AutoLoadBalancedConnectionStrategy [42092 TcpOutEloop] - Applying quarantine to ip=A.B.C.D port=9997 _numberOfFailures=2
07-23-2021 11:45:42.188 +0000 WARN TcpOutputProc [42091 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=A.B.C.D inside output group default-autolb-group from host_src=UF_name has been blocked for blocked_seconds=3000. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
07-23-2021 11:47:22.196 +0000 WARN TcpOutputProc [42091 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=A.B.C.D inside output group default-autolb-group from host_src=UF_name has been blocked for blocked_seconds=3100. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
07-23-2021 11:49:02.204 +0000 WARN TcpOutputProc [42091 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=A.B.C.D inside output group default-autolb-group from host_src=UF_name has been blocked for blocked_seconds=3200. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
07-23-2021 11:50:29.730 +0000 INFO AutoLoadBalancedConnectionStrategy [42092 TcpOutEloop] - Removing quarantine from idx=A.B.C.D:9997
07-23-2021 11:50:29.732 +0000 ERROR TcpOutputFd [42092 TcpOutEloop] - Read error. Connection reset by peer
07-23-2021 11:50:29.734 +0000 ERROR TcpOutputFd [42092 TcpOutEloop] - Read error. Connection reset by peer
07-23-2021 11:50:29.734 +0000 WARN AutoLoadBalancedConnectionStrategy [42092 TcpOutEloop] - Applying quarantine to ip=A.B.C.D port=9997 _numberOfFailures=2

 

tcpdump also showed me reset from HF side. 

 

I have communication between UF and HF - all necessary ports are open. 

[root@UF_name ~]# nc -z -v A.B.C.D 9997
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to A.B.C.D:9997.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
[root@UF_name ~]# nc -z -v A.B.C.D 8000
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to A.B.C.D6:8000.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
[root@UF_name ~]# nc -z -v A.B.C.D 8089
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to A.B.C.D:8089.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.

 

How to solve this problem? Any tips?

 

Labels (4)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Check the logs on the HF side to see if it explains why the HF is dropping the connections.

If you're using SSL, verify the certificates are correct on both sides.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

dabroma5
Explorer

Hi

Do you mean splunkd.log? 

SSL is only used towards cloud

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!