Why am I unable to start 2 of 4 search heads with "WARNING: web interface does not seem to be available!"?

[jamesoconnell]

I have four search head instances running in our NONPROD environment, but am getting the following error below when starting two of the four instances.

We are running Splunk Enterprise Splunk 6.2.0 (build 237341)

Regards,
-James O'Connell

...

 All preliminary checks passed.
 Starting splunk server daemon (splunkd)...
 Done
                                                       [  OK  ]
 Waiting for web server at http://127.0.0.1:38000 to be available
 WARNING: web interface does not seem to be available!

[sloshburch]

When I've had this issue, I chalked it up to entropy on the system over time. Assuming the config and the ports are all good, then I start with some simple and basic things:

1. Run splunk stop and then do a kill -9 on the results of: ps -ef | grep -i splunk | grep -v grep | awk '{print $2;}' to stop anything that didn't actually shut down
2. Restart the actual server thereby purging any zombie processes that might not be needed but are still running from usage over time
3. When all else fails, reinstall that instance. This can be made easy through proper use of the configuration distribution features

[renjith_nair]

Are you running all instances on same machine? If that's the case make sure your ports are not conflicting.

Check splunkd.log for errors. One of your configuration might be incorrect or there are some conflicts.

Also check if mongod instance (kvstore) is shutdown properly and no process is running. We had an issue where stale process of mongo was running which prevents re-starting.

As always, log files are the first place to look for any issues.