Got Error: "The splunk daemon (splunkd) is already running. " when restarting splunk.

[kamlesh_vaghela]

Hello Team,

Well, I'm facing strange error when restarting splunk.

# /opt/splunk/bin/splunk restart
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
...                                                        [  OK  ]
Stopping splunk helpers...
                                                           [  OK  ]
Done.
The splunk daemon (splunkd) is already running.            [FAILED]

Waiting for web server at http://127.0.0.1:8000 to be available.. Done


If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com

The Splunk web interface is at http://sh:8000

Please see error: The splunk daemon (splunkd) is already running. [FAILED].

When I restarted again immediately then it will restart normally.

My question is if splunk is Stopping splunkd... then which kind of process is still running which cause The splunk daemon (splunkd) is already running. [FAILED] ???

An information/clue /idea will be appreciated.

Thanks

[nickhills]

This occurs when splunk has stopped running (uncleanly), but it leaves a pid file behind.

During the restart, Splunk checks to see if there is an exiting splunk pid, and since it finds one it reports:
The splunk daemon (splunkd) is already running.

(but its not!)
Since you have requested a restart, it tries to shut down the process defined in the pid, but of course it no longer exists, so it reports
[FAILED]

It will then start up normally, and if you restart again, it will work as expected.

The sequence of "The splunk daemon....[Failed]" is not in itself anything to worry about, however I would be a bit concerned about what happened to kill the process in the first place.

[mayurr98]

hey @kamlesh

Try this:

rm splunkforwarder/var/run/splunk/splunkd.pid

let me know if it helps you!

[DavidHourani]

Hi Kamlesh

Try running a splunk stop followed by a splunk start to see if you get the same outcome.
If so stop splunk again and run: ps aux | grep splunkd | grep -v grep | grep -v search
Also check your splunkd.log file for any errors/warnings during the restart and paste them here so we can look further at what can be the problem.

Cheers,
David

[harsmarvania57]

It looks like when you stop splunkd process, it didn't exited cleanly and immediately try to start splunkd again which throws an error.

Next time when you will see this error at that time before restarting splunk again check processes on server with command ps -ef | grep splunk