Well, I'm facing strange error when restarting splunk.
# /opt/splunk/bin/splunk restart Stopping splunkd... Shutting down. Please wait, as this may take a few minutes. ... [ OK ] Stopping splunk helpers... [ OK ] Done. The splunk daemon (splunkd) is already running. [FAILED] Waiting for web server at http://127.0.0.1:8000 to be available.. Done If you get stuck, we're here to help. Look for answers here: http://docs.splunk.com The Splunk web interface is at http://sh:8000
Please see error:
The splunk daemon (splunkd) is already running. [FAILED].
When I restarted again immediately then it will restart normally.
My question is if splunk is
Stopping splunkd... then which kind of process is still running which cause
The splunk daemon (splunkd) is already running. [FAILED] ???
An information/clue /idea will be appreciated.
This occurs when splunk has stopped running (uncleanly), but it leaves a pid file behind.
During the restart, Splunk checks to see if there is an exiting splunk pid, and since it finds one it reports:
The splunk daemon (splunkd) is already running.
(but its not!)
Since you have requested a restart, it tries to shut down the process defined in the pid, but of course it no longer exists, so it reports
It will then start up normally, and if you restart again, it will work as expected.
The sequence of "The splunk daemon....[Failed]" is not in itself anything to worry about, however I would be a bit concerned about what happened to kill the process in the first place.
Try running a splunk stop followed by a splunk start to see if you get the same outcome.
If so stop splunk again and run: ps aux | grep splunkd | grep -v grep | grep -v search
Also check your splunkd.log file for any errors/warnings during the restart and paste them here so we can look further at what can be the problem.
It looks like when you stop splunkd process, it didn't exited cleanly and immediately try to start splunkd again which throws an error.
Next time when you will see this error at that time before restarting splunk again check processes on server with command
ps -ef | grep splunk