Monitoring Splunk

Timed out waiting for splunkd to shutdown

kenison
New Member

I can't restart splunk. Neither CLI or web manager works. It just times out waiting for splunkd to shut down. Thing is, if I kill -9 everything splunk, it still thinks it needs to stop splunkd and splunkweb. And If I kill everything and try to start, it thinks splunkd is still running. This is really frustrating.

Tags (1)
0 Karma

jmccallhbo
Explorer

I am having a similar issue. We are starting Splunk with a golang script that runs the command /opt/splunk/bin/splunk start. This leaves behind a zombie splunkd process but otherwise starts fine. Running splunk stop will time out and it prints that it timed out to the console. Interestingly splunk does stop in less than 2 secs and the splunk.pid file is removed. What is splunk stop waiting for???

0 Karma

Simeon
Splunk Employee
Splunk Employee

Sounds like splunk is hung trying to execute or close an activity. When running forwarders, Splunk will close these connections to the indexer gracefully. In some cases, Splunk might be still writing to disk and is waiting for that activity to complete.

The best thing to do is look at the $SPLUNK_HOME/var/log/splunkd.log file and check for any errors during shutdown. Additionaly, splunk creates *.pid files that map if Splunk is still running or not. If those pid files still exist, Splunk will not start. In that case, you could manually remove them.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...