Splunk forwarder starts, then throws an error saying splunk hasn't started.

[bcurtiss]

I'm trying to get a splunk forwarder running on a linux box, but when I try to tell the forwarder to forward to a specific indexer, it throws an error saying that Splunk is not running. Anyone ever have this issue? I'm trying to use version 4.3.5.

This is what I tried to do:

[root:/opt/splunkforwarder/bin]# ./splunk start

Splunk> Finding your faults, just like mom.

Checking prerequisites...
Checking mgmt port [8089]: open
Checking conf files for typos...
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
[ OK ]
Done.[root:/opt/splunkforwarder/bin]# ./splunk add localhost myindexer.net:9997
Splunk is not running, and it must be for this operation. To start splunk, run "splunk start".
[root:/opt/splunkforwarder/bin]#

[bcurtiss · Accepted solution]

Thanks for the responses, but the problem was actually just because of a crappy firewall rule. I ended up running an strace and saw that it was failing on connect(); I fixed the rule and everything is fine now.

[tbarnard]

I've seen that behavior before when there old PID file is stuck. With splunk stopped check and see if /opt/splunkforwarder/var/run/splunk/splunkd.pid is still there and delete it if it is. Then start splunk again.

[tbarnard]

Have you started splunkforwarder before? The first time splunk starts you will need to accept it's license.

[yannK]

try ./splunk status to see what is the process doing.
and check the internal logs in $SPLUNK_HOME/var/log/splunk/splunkd.log for errors.