"splunk restart" command takes long time

[melonman]

Hi there,

When I issued "splunk restart" command, it takes more than 5 min.
Looks like stopping splunk takes most of the restart time.

Could you give us the possible reasons why restarting splunk takes longer time?

Thank you!

[jbsplunk · Accepted solution]

When you stop Splunk, it has to close out all the inputs, as well as the indexes and other components of the product. The more of these you've got floating around, the longer it takes. Splunkd.log in $SPLUNK_HOME/var/log/splunk/ will tell you what is shutting down when the shutdown occurs. If your particularly curious what is taking so long, and you can't tell from splunkd.log, you can probably strace splunkd during the shutdown to see what is happening. For Windows, the equivalent tool would be procmon. I think you'll find the answer here.

[lguinn2]

I have found that stopping Splunk takes longer when there are searches running, especially realtime searches. My guess is that Splunk sends a "stop" to each running subprocess, and then waits a bit to allow them to stop gracefully. How does that relate to your experience? Do you have many users running searches? Does Splunk stop faster if you exit the UI and do the stop command from the command line?

[Drainy]

are there any errors displayed in the splunkd.log around the time of the shutdown? Might be worth installing the SoS app and using it to have a look at your internal logs for errors or warnings (perhaps even crashes on shutdown)