I have Splunk working on one server (an indexer) with one other server as its client (with the Universal forwarder). All my machines are Linux. I want to get Splunk to work with an additional client.
It seems like port 9997 is closed on my network. At this time of year, I cannot get someone to determine if it is open or not. iptables doesn't block this port on either machine (the client forwarder that I want to get working or the Splunk server). I installed telnet on both machines.
On the forwarder I want to get working for the first time, the output of this command (from /opt/splunkforwarder/bin/) is nothing:
# ./splunk cmd btool output list --debug
The output of this command from /opt/splunkforwarder/bin/ (from a client server that is not yet a forwarder),
# ./splunk cmd btool inputs list splunktcp --debug
is as follows:
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf [splunktcp] /opt/splunkforwarder/etc/system/default/inputs.conf _rcvbuf = 1572864 /opt/splunkforwarder/etc/system/default/inputs.conf acceptFrom = * /opt/splunkforwarder/etc/system/default/inputs.conf connection_host = ip /opt/splunkforwarder/etc/system/local/inputs.conf host = cooltest.domainName.cloud /opt/splunkforwarder/etc/system/default/inputs.conf index = default /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf route = has_key:tautology:parsingQueue;absent_key:tautology:parsingQueue
On the main Splunk server, I did a tail of the splunkd.log file. I found this:
12-31-2014 16:12:28.663 -0800 ERROR TcpOutputFd - Connection to host=x.x.x.x:80 failed 12-31-2014 16:12:58.665 -0800 WARN TcpOutputFd - Connect to x.x.x.x:80 failed. Connection refused
Where x.x.x.x is the IP address of the client server that I want to forward. nmap showed that port 80 was blocked between the servers.
On the client server (that I want to be a forwarder), I did a tail of the splunkd.log file. I found this:
01-01-2015 00:16:47.426 +0000 ERROR TcpOutputFd - Connection to host=y.y.y.y:9997 failed 01-01-2015 00:16:48.429 +0000 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 9600 seconds. 01-01-2015 00:17:17.428 +0000 WARN TcpOutputFd - Connect to y.y.y.y:9997 failed. Connection refused
Where y.y.y.y is the IP address of main Splunk server.
What should I do to get Splunk working with this client server? I want the client server to be a forwarder.
The first step would be to run from
the client the following - telnet
'splunk server host' 9997
I get this:
Trying x.x.x.x... telnet: connect to address x.x.x.x: Connection refused
where x.x.x.x is the IP address of the main Splunk server (aka the indexer).