topic Re: "splunk restart" command takes long time in Deployment Architecture

"splunk restart" command takes long time

melonman — Wed, 21 Dec 2011 00:02:00 GMT

Hi there,

When I issued "splunk restart" command, it takes more than 5 min.
Looks like stopping splunk takes most of the restart time.

Could you give us the possible reasons why restarting splunk takes longer time?

Thank you!

Re: "splunk restart" command takes long time

Drainy — Wed, 21 Dec 2011 09:44:21 GMT

are there any errors displayed in the splunkd.log around the time of the shutdown? Might be worth installing the SoS app and using it to have a look at your internal logs for errors or warnings (perhaps even crashes on shutdown)

Re: "splunk restart" command takes long time

lguinn2 — Wed, 21 Dec 2011 21:50:48 GMT

I have found that stopping Splunk takes longer when there are searches running, especially realtime searches. My guess is that Splunk sends a "stop" to each running subprocess, and then waits a bit to allow them to stop gracefully. How does that relate to your experience? Do you have many users running searches? Does Splunk stop faster if you exit the UI and do the stop command from the command line?

Re: "splunk restart" command takes long time

jbsplunk — Wed, 21 Dec 2011 23:54:06 GMT

When you stop Splunk, it has to close out all the inputs, as well as the indexes and other components of the product. The more of these you've got floating around, the longer it takes. Splunkd.log in $SPLUNK_HOME/var/log/splunk/ will tell you what is shutting down when the shutdown occurs. If your particularly curious what is taking so long, and you can't tell from splunkd.log, you can probably strace splunkd during the shutdown to see what is happening. For Windows, the equivalent tool would be procmon. I think you'll find the answer here.