Getting Data In

WARN TcpOutputFd - Connect to host:port failed. Connection refused

splunker12er
Motivator

I am forwarding data from heavy-forwarder (HF-1) to heavy-forwarder(HF-2) which are in different network IP range.

Eg:
10.172.0.1 to 10.234.0.1

I have enabled the forwarding from HF-1 to HF-2 via TCP/9999 port.

outputs.conf (HF-1) :forwarding-end

[tcpout]
defaultGroup = default-autolb-group
indexAndForward = 0

[tcpout:default-autolb-group]
disabled = 0
server = 10.234.0.1:9999

[tcpout-server://10.234.0.1:9999]

inputs.conf in HF-2 : (receiving-end) under launcher app

[splunktcp://9999]
connection_host = none

splunkd.logs:
11-20-2015 10:26:41.868 +0000 WARN TcpOutputFd - Connect to 10.234.0.1:9999 failed. Connection refused
11-20-2015 10:26:41.868 +0000 ERROR TcpOutputFd - Connection to host=10.234.0.1:9999 failed
11-20-2015 10:26:41.868 +0000 WARN TcpOutputProc - Applying quarantine to ip=10.234.0.1 port=9999 _numberOfFailures=2

network troubleshooting:

At HF-1
Telnet to HF-2 from HF-1 for 9999 port

telnet 10.234.0.1 9999
-- which gets connected for the first time..
But after sometime failed to connect

At HF-2:

netstat -anp|grep 9999

bash-4.1$ netstat -anp|grep 9999
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp   138835      0 10.234.0.1:9999            10.234.0.1:49679          ESTABLISHED 18110/splunkd 
0 Karma

ltrand
Contributor

How many events per minute are each handling, and HF-2 specifically. Also, how many forwarders total is HF02 handling? HF02 is refusing to allow other connections to come through, or one/many of its queues are filling up and it's telling HF01 to stop momentarily. If you can provide more information about your environment then a better answer can be provided.

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Laser Bananas and Edge Hubs: Exploring Operational Technology (OT) Data Through a ...

  OT is a different environment to traditional IT and can have interesting challenges when interfacing the ...

Event Series: Mastering AI Tokenomics and Splunk Agent Observability

Beyond the Black Box: Correlating AI Performance and Tokenomics with Splunk Agent Observability   As ...

span_metrics: The OpenTelemetry-Idiomatic Way to See Inside Your Services

You open a trace in Splunk Observability Cloud and everything looks fine. One root span, order-pipeline, with ...