Monitoring Splunk

Got Error: "The splunk daemon (splunkd) is already running. " when restarting splunk.

kamlesh_vaghela
SplunkTrust
SplunkTrust

Hello Team,

Well, I'm facing strange error when restarting splunk.

# /opt/splunk/bin/splunk restart
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
...                                                        [  OK  ]
Stopping splunk helpers...
                                                           [  OK  ]
Done.
The splunk daemon (splunkd) is already running.            [FAILED]

Waiting for web server at http://127.0.0.1:8000 to be available.. Done


If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com

The Splunk web interface is at http://sh:8000

Please see error: The splunk daemon (splunkd) is already running. [FAILED].

When I restarted again immediately then it will restart normally.

My question is if splunk is Stopping splunkd... then which kind of process is still running which cause The splunk daemon (splunkd) is already running. [FAILED] ???

An information/clue /idea will be appreciated.

Thanks

Tags (1)
0 Karma

nickhills
Ultra Champion

This occurs when splunk has stopped running (uncleanly), but it leaves a pid file behind.

During the restart, Splunk checks to see if there is an exiting splunk pid, and since it finds one it reports:
The splunk daemon (splunkd) is already running.

(but its not!)
Since you have requested a restart, it tries to shut down the process defined in the pid, but of course it no longer exists, so it reports
[FAILED]

It will then start up normally, and if you restart again, it will work as expected.

The sequence of "The splunk daemon....[Failed]" is not in itself anything to worry about, however I would be a bit concerned about what happened to kill the process in the first place.

If my comment helps, please give it a thumbs up!

mayurr98
Super Champion

hey @kamlesh

Try this:

rm splunkforwarder/var/run/splunk/splunkd.pid

let me know if it helps you!

0 Karma

DavidHourani
Super Champion

Hi Kamlesh

Try running a splunk stop followed by a splunk start to see if you get the same outcome.
If so stop splunk again and run: ps aux | grep splunkd | grep -v grep | grep -v search
Also check your splunkd.log file for any errors/warnings during the restart and paste them here so we can look further at what can be the problem.

Cheers,
David

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

It looks like when you stop splunkd process, it didn't exited cleanly and immediately try to start splunkd again which throws an error.

Next time when you will see this error at that time before restarting splunk again check processes on server with command ps -ef | grep splunk

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...