Getting Data In

WARN TcpOutputFd - Connect to host:port failed. Connection refused

splunker12er
Motivator

I am forwarding data from heavy-forwarder (HF-1) to heavy-forwarder(HF-2) which are in different network IP range.

Eg:
10.172.0.1 to 10.234.0.1

I have enabled the forwarding from HF-1 to HF-2 via TCP/9999 port.

outputs.conf (HF-1) :forwarding-end

[tcpout]
defaultGroup = default-autolb-group
indexAndForward = 0

[tcpout:default-autolb-group]
disabled = 0
server = 10.234.0.1:9999

[tcpout-server://10.234.0.1:9999]

inputs.conf in HF-2 : (receiving-end) under launcher app

[splunktcp://9999]
connection_host = none

splunkd.logs:
11-20-2015 10:26:41.868 +0000 WARN TcpOutputFd - Connect to 10.234.0.1:9999 failed. Connection refused
11-20-2015 10:26:41.868 +0000 ERROR TcpOutputFd - Connection to host=10.234.0.1:9999 failed
11-20-2015 10:26:41.868 +0000 WARN TcpOutputProc - Applying quarantine to ip=10.234.0.1 port=9999 _numberOfFailures=2

network troubleshooting:

At HF-1
Telnet to HF-2 from HF-1 for 9999 port

telnet 10.234.0.1 9999
-- which gets connected for the first time..
But after sometime failed to connect

At HF-2:

netstat -anp|grep 9999

bash-4.1$ netstat -anp|grep 9999
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp   138835      0 10.234.0.1:9999            10.234.0.1:49679          ESTABLISHED 18110/splunkd 
0 Karma

ltrand
Contributor

How many events per minute are each handling, and HF-2 specifically. Also, how many forwarders total is HF02 handling? HF02 is refusing to allow other connections to come through, or one/many of its queues are filling up and it's telling HF01 to stop momentarily. If you can provide more information about your environment then a better answer can be provided.

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Quantify Your Splunk Investment Impact: Introducing Savings Metrics to Value Insights

Building on the foundation established in our initial Value Insights releases, we are introducing the Savings ...

Event Series: Telemetry Pipeline Management

Balancing Scale and Spend: Gaining Control Over High-Volume Metrics in Splunk Observability Cloud As ...

Kick the Tires Before You Commit: A Hands-On Tour of the Splunk Observability Cloud ...

Evaluating an enterprise observability platform usually goes like this: fill out a form, get a free trial with ...