I have four search head instances running in our NONPROD environment, but am getting the following error below when starting two of the four instances.
We are running Splunk Enterprise Splunk 6.2.0 (build 237341)
Regards,
-James O'Connell
...
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
Done
[ OK ]
Waiting for web server at http://127.0.0.1:38000 to be available
WARNING: web interface does not seem to be available!
When I've had this issue, I chalked it up to entropy on the system over time. Assuming the config and the ports are all good, then I start with some simple and basic things:
splunk stop
and then do a kill -9
on the results of: ps -ef | grep -i splunk | grep -v grep | awk '{print $2;}'
to stop anything that didn't actually shut downAre you running all instances on same machine? If that's the case make sure your ports are not conflicting.
Check splunkd.log for errors. One of your configuration might be incorrect or there are some conflicts.
Also check if mongod instance (kvstore) is shutdown properly and no process is running. We had an issue where stale process of mongo was running which prevents re-starting.
As always, log files are the first place to look for any issues.