Monitoring Splunk

Timed out waiting for splunkd to shutdown

New Member

I can't restart splunk. Neither CLI or web manager works. It just times out waiting for splunkd to shut down. Thing is, if I kill -9 everything splunk, it still thinks it needs to stop splunkd and splunkweb. And If I kill everything and try to start, it thinks splunkd is still running. This is really frustrating.

Tags (1)
0 Karma


I am having a similar issue. We are starting Splunk with a golang script that runs the command /opt/splunk/bin/splunk start. This leaves behind a zombie splunkd process but otherwise starts fine. Running splunk stop will time out and it prints that it timed out to the console. Interestingly splunk does stop in less than 2 secs and the file is removed. What is splunk stop waiting for???

0 Karma

Splunk Employee
Splunk Employee

Sounds like splunk is hung trying to execute or close an activity. When running forwarders, Splunk will close these connections to the indexer gracefully. In some cases, Splunk might be still writing to disk and is waiting for that activity to complete.

The best thing to do is look at the $SPLUNK_HOME/var/log/splunkd.log file and check for any errors during shutdown. Additionaly, splunk creates *.pid files that map if Splunk is still running or not. If those pid files still exist, Splunk will not start. In that case, you could manually remove them.

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!