Getting Data In

WARN TcpOutputFd - Connect to host:port failed. Connection refused

splunker12er
Motivator

I am forwarding data from heavy-forwarder (HF-1) to heavy-forwarder(HF-2) which are in different network IP range.

Eg:
10.172.0.1 to 10.234.0.1

I have enabled the forwarding from HF-1 to HF-2 via TCP/9999 port.

outputs.conf (HF-1) :forwarding-end

[tcpout]
defaultGroup = default-autolb-group
indexAndForward = 0

[tcpout:default-autolb-group]
disabled = 0
server = 10.234.0.1:9999

[tcpout-server://10.234.0.1:9999]

inputs.conf in HF-2 : (receiving-end) under launcher app

[splunktcp://9999]
connection_host = none

splunkd.logs:
11-20-2015 10:26:41.868 +0000 WARN TcpOutputFd - Connect to 10.234.0.1:9999 failed. Connection refused
11-20-2015 10:26:41.868 +0000 ERROR TcpOutputFd - Connection to host=10.234.0.1:9999 failed
11-20-2015 10:26:41.868 +0000 WARN TcpOutputProc - Applying quarantine to ip=10.234.0.1 port=9999 _numberOfFailures=2

network troubleshooting:

At HF-1
Telnet to HF-2 from HF-1 for 9999 port

telnet 10.234.0.1 9999
-- which gets connected for the first time..
But after sometime failed to connect

At HF-2:

netstat -anp|grep 9999

bash-4.1$ netstat -anp|grep 9999
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp   138835      0 10.234.0.1:9999            10.234.0.1:49679          ESTABLISHED 18110/splunkd 
0 Karma

ltrand
Contributor

How many events per minute are each handling, and HF-2 specifically. Also, how many forwarders total is HF02 handling? HF02 is refusing to allow other connections to come through, or one/many of its queues are filling up and it's telling HF01 to stop momentarily. If you can provide more information about your environment then a better answer can be provided.

Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...