How to troubleshoot error "Connection to host=xx.xxx.x.xxx:9997 failed"?

[johnbuhlhiscox]

Splunk events are stopping. When we check the logs, we are seeing all of these errors. When we restart Splunk, it works for a few minutes and then goes away again. Here is the error message.

06-21-2016 22:20:11.986 +0100 ERROR TcpOutputFd - Connection to host=xx.xxx.x.xxx:9997 failed
06-21-2016 22:20:19.396 +0100 WARN  TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 1300 seconds.
06-21-2016 22:20:41.969 +0100 WARN  TcpOutputFd - Connect to xx.xxx.x.xxx:9997 failed. No connection could be made because the target machine actively refused it.
06-21-2016 22:20:41.969 +0100 ERROR TcpOutputFd - Connection to host=xx.xxx.x.xxx:9997 failed
06-21-2016 22:21:11.982 +0100 WARN  TcpOutputFd - Connect to xx.xxx.x.xxx:9997 failed. No connection could be made because the target machine actively refused it.
06-21-2016 22:21:11.982 +0100 ERROR TcpOutputFd - Connection to host=xx.xxx.x.xxx:9997 failed
06-21-2016 22:21:41.984 +0100 WARN  TcpOutputFd - Connect to xx.xxx.x.xxx:9997 failed. No connection could be made because the target machine actively refused it.
06-21-2016 22:21:41.984 +0100 ERROR TcpOutputFd - Connection to host=xx.xxx.x.xxx:9997 failed
06-21-2016 22:21:59.507 +0100 WARN  TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 1400 seconds.
06-21-2016 22:22:11.974 +0100 WARN  TcpOutputFd - Connect to xx.xxx.x.xxx:9997 failed. No connection could be made because the target machine actively refused it.
06-21-2016 22:22:11.975 +0100 ERROR TcpOutputFd - Connection to host=xx.xxx.x.xxx:9997 failed

We have tried what previous users suggested by adding this to the inputs.conf to no avail.

[splunktcp://9997]
connection_host = none

[dkingfx]

Make sure you have this line in your iptable
cat /etc/sysconfig/iptables
vi /etc/sysconfig/iptables
-A INPUT -m state --state NEW -m tcp -p tcp --dport 9997 -j ACCEPT

[woodcock]

That is the default port for the forwarders to send (non-SSL) data to the Indexers:
https://answers.splunk.com/answers/118859/diagram-of-splunk-common-network-ports.html

Unless you changed something on purpose, it should "just work" right out of the box, if you have port 9997 opened for incoming connections from you forwarders to your indexers. This looks like a classic firewall deny scenario.