Why isn't Splunk working with a new forwarder client?

[AllenRed]

I have Splunk working on one server (an indexer) with one other server as its client (with the Universal forwarder). All my machines are Linux. I want to get Splunk to work with an additional client.

It seems like port 9997 is closed on my network. At this time of year, I cannot get someone to determine if it is open or not. iptables doesn't block this port on either machine (the client forwarder that I want to get working or the Splunk server). I installed telnet on both machines.

On the forwarder I want to get working for the first time, the output of this command (from /opt/splunkforwarder/bin/) is nothing:

 # ./splunk cmd btool output list --debug 

The output of this command from /opt/splunkforwarder/bin/ (from a client server that is not yet a forwarder),

 # ./splunk cmd btool inputs list splunktcp --debug

is as follows:

 /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf [splunktcp]
 /opt/splunkforwarder/etc/system/default/inputs.conf                        _rcvbuf = 1572864
 /opt/splunkforwarder/etc/system/default/inputs.conf                        acceptFrom = *
 /opt/splunkforwarder/etc/system/default/inputs.conf                        connection_host = ip
 /opt/splunkforwarder/etc/system/local/inputs.conf                          host = cooltest.domainName.cloud
 /opt/splunkforwarder/etc/system/default/inputs.conf                        index = default
 /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf route = has_key:tautology:parsingQueue;absent_key:tautology:parsingQueue

On the main Splunk server, I did a tail of the splunkd.log file. I found this:

12-31-2014 16:12:28.663 -0800 ERROR TcpOutputFd - Connection to host=x.x.x.x:80 failed
12-31-2014 16:12:58.665 -0800 WARN  TcpOutputFd - Connect to x.x.x.x:80 failed. Connection refused

Where x.x.x.x is the IP address of the client server that I want to forward. nmap showed that port 80 was blocked between the servers.

On the client server (that I want to be a forwarder), I did a tail of the splunkd.log file. I found this:

01-01-2015 00:16:47.426 +0000 ERROR TcpOutputFd - Connection to host=y.y.y.y:9997 failed
01-01-2015 00:16:48.429 +0000 WARN  TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 9600 seconds.
01-01-2015 00:17:17.428 +0000 WARN  TcpOutputFd - Connect to y.y.y.y:9997 failed. Connection refused

Where y.y.y.y is the IP address of main Splunk server.

What should I do to get Splunk working with this client server? I want the client server to be a forwarder.

[ddrillic]

No good – no connectivity ... did you put the port as well in the telnet command?

[AllenRed]

The first step would be to run from
the client the following - telnet
'splunk server host' 9997

I get this:

Trying x.x.x.x...
telnet: connect to address x.x.x.x: Connection refused

where x.x.x.x is the IP address of the main Splunk server (aka the indexer).

[ddrillic]

The first step would be to run from the client the following -
telnet 'splunk server host' 9997

Regards,
Dan