Getting Data In

WARN TcpOutputFd - Connect to host:port failed. Connection refused


I am forwarding data from heavy-forwarder (HF-1) to heavy-forwarder(HF-2) which are in different network IP range.

Eg: to

I have enabled the forwarding from HF-1 to HF-2 via TCP/9999 port.

outputs.conf (HF-1) :forwarding-end

defaultGroup = default-autolb-group
indexAndForward = 0

disabled = 0
server =


inputs.conf in HF-2 : (receiving-end) under launcher app

connection_host = none

11-20-2015 10:26:41.868 +0000 WARN TcpOutputFd - Connect to failed. Connection refused
11-20-2015 10:26:41.868 +0000 ERROR TcpOutputFd - Connection to host= failed
11-20-2015 10:26:41.868 +0000 WARN TcpOutputProc - Applying quarantine to ip= port=9999 _numberOfFailures=2

network troubleshooting:

At HF-1
Telnet to HF-2 from HF-1 for 9999 port

telnet 9999
-- which gets connected for the first time..
But after sometime failed to connect

At HF-2:

netstat -anp|grep 9999

bash-4.1$ netstat -anp|grep 9999
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp   138835      0            ESTABLISHED 18110/splunkd 
0 Karma

Re: WARN TcpOutputFd - Connect to host:port failed. Connection refused


How many events per minute are each handling, and HF-2 specifically. Also, how many forwarders total is HF02 handling? HF02 is refusing to allow other connections to come through, or one/many of its queues are filling up and it's telling HF01 to stop momentarily. If you can provide more information about your environment then a better answer can be provided.