Getting Data In

WARN TcpOutputFd - Connect to host:port failed. Connection refused

splunker12er
Motivator

I am forwarding data from heavy-forwarder (HF-1) to heavy-forwarder(HF-2) which are in different network IP range.

Eg:
10.172.0.1 to 10.234.0.1

I have enabled the forwarding from HF-1 to HF-2 via TCP/9999 port.

outputs.conf (HF-1) :forwarding-end

[tcpout]
defaultGroup = default-autolb-group
indexAndForward = 0

[tcpout:default-autolb-group]
disabled = 0
server = 10.234.0.1:9999

[tcpout-server://10.234.0.1:9999]

inputs.conf in HF-2 : (receiving-end) under launcher app

[splunktcp://9999]
connection_host = none

splunkd.logs:
11-20-2015 10:26:41.868 +0000 WARN TcpOutputFd - Connect to 10.234.0.1:9999 failed. Connection refused
11-20-2015 10:26:41.868 +0000 ERROR TcpOutputFd - Connection to host=10.234.0.1:9999 failed
11-20-2015 10:26:41.868 +0000 WARN TcpOutputProc - Applying quarantine to ip=10.234.0.1 port=9999 _numberOfFailures=2

network troubleshooting:

At HF-1
Telnet to HF-2 from HF-1 for 9999 port

telnet 10.234.0.1 9999
-- which gets connected for the first time..
But after sometime failed to connect

At HF-2:

netstat -anp|grep 9999

bash-4.1$ netstat -anp|grep 9999
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp   138835      0 10.234.0.1:9999            10.234.0.1:49679          ESTABLISHED 18110/splunkd 
0 Karma

ltrand
Contributor

How many events per minute are each handling, and HF-2 specifically. Also, how many forwarders total is HF02 handling? HF02 is refusing to allow other connections to come through, or one/many of its queues are filling up and it's telling HF01 to stop momentarily. If you can provide more information about your environment then a better answer can be provided.

Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...