Monitoring Splunk
Provide Splunk Cloud feedback in this confidential UX survey by June 17
for a chance to win a $200 Amazon gift card!

Timed out waiting for splunkd to shutdown

kenison
New Member

I can't restart splunk. Neither CLI or web manager works. It just times out waiting for splunkd to shut down. Thing is, if I kill -9 everything splunk, it still thinks it needs to stop splunkd and splunkweb. And If I kill everything and try to start, it thinks splunkd is still running. This is really frustrating.

Tags (1)
0 Karma

jmccallhbo
Explorer

I am having a similar issue. We are starting Splunk with a golang script that runs the command /opt/splunk/bin/splunk start. This leaves behind a zombie splunkd process but otherwise starts fine. Running splunk stop will time out and it prints that it timed out to the console. Interestingly splunk does stop in less than 2 secs and the splunk.pid file is removed. What is splunk stop waiting for???

0 Karma

Simeon
Splunk Employee
Splunk Employee

Sounds like splunk is hung trying to execute or close an activity. When running forwarders, Splunk will close these connections to the indexer gracefully. In some cases, Splunk might be still writing to disk and is waiting for that activity to complete.

The best thing to do is look at the $SPLUNK_HOME/var/log/splunkd.log file and check for any errors during shutdown. Additionaly, splunk creates *.pid files that map if Splunk is still running or not. If those pid files still exist, Splunk will not start. In that case, you could manually remove them.

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!