Installation

UF to HF - no active forwards

dabroma5
Explorer

Hi,

I am trying to configure Universal Forwarder and Heavy forwarder.

In UF  I see:

Active forwards:
None
Configured but inactive forwards:
A.B.C.D:9997

splunkd.log:

07-23-2021 11:45:00.807 +0000 WARN AutoLoadBalancedConnectionStrategy [42092 TcpOutEloop] - Applying quarantine to ip=A.B.C.D port=9997 _numberOfFailures=2
07-23-2021 11:45:42.188 +0000 WARN TcpOutputProc [42091 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=A.B.C.D inside output group default-autolb-group from host_src=UF_name has been blocked for blocked_seconds=3000. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
07-23-2021 11:47:22.196 +0000 WARN TcpOutputProc [42091 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=A.B.C.D inside output group default-autolb-group from host_src=UF_name has been blocked for blocked_seconds=3100. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
07-23-2021 11:49:02.204 +0000 WARN TcpOutputProc [42091 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=A.B.C.D inside output group default-autolb-group from host_src=UF_name has been blocked for blocked_seconds=3200. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
07-23-2021 11:50:29.730 +0000 INFO AutoLoadBalancedConnectionStrategy [42092 TcpOutEloop] - Removing quarantine from idx=A.B.C.D:9997
07-23-2021 11:50:29.732 +0000 ERROR TcpOutputFd [42092 TcpOutEloop] - Read error. Connection reset by peer
07-23-2021 11:50:29.734 +0000 ERROR TcpOutputFd [42092 TcpOutEloop] - Read error. Connection reset by peer
07-23-2021 11:50:29.734 +0000 WARN AutoLoadBalancedConnectionStrategy [42092 TcpOutEloop] - Applying quarantine to ip=A.B.C.D port=9997 _numberOfFailures=2

 

tcpdump also showed me reset from HF side. 

 

I have communication between UF and HF - all necessary ports are open. 

[root@UF_name ~]# nc -z -v A.B.C.D 9997
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to A.B.C.D:9997.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
[root@UF_name ~]# nc -z -v A.B.C.D 8000
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to A.B.C.D6:8000.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
[root@UF_name ~]# nc -z -v A.B.C.D 8089
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to A.B.C.D:8089.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.

 

How to solve this problem? Any tips?

 

Labels (4)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Check the logs on the HF side to see if it explains why the HF is dropping the connections.

If you're using SSL, verify the certificates are correct on both sides.

---
If this reply helps you, Karma would be appreciated.
0 Karma

dabroma5
Explorer

Hi

Do you mean splunkd.log? 

SSL is only used towards cloud

0 Karma
Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...