Deployment Architecture

"splunk restart" command takes long time

melonman
Motivator

Hi there,

When I issued "splunk restart" command, it takes more than 5 min.
Looks like stopping splunk takes most of the restart time.

Could you give us the possible reasons why restarting splunk takes longer time?

Thank you!

Tags (1)
1 Solution

jbsplunk
Splunk Employee
Splunk Employee

When you stop Splunk, it has to close out all the inputs, as well as the indexes and other components of the product. The more of these you've got floating around, the longer it takes. Splunkd.log in $SPLUNK_HOME/var/log/splunk/ will tell you what is shutting down when the shutdown occurs. If your particularly curious what is taking so long, and you can't tell from splunkd.log, you can probably strace splunkd during the shutdown to see what is happening. For Windows, the equivalent tool would be procmon. I think you'll find the answer here.

View solution in original post

jbsplunk
Splunk Employee
Splunk Employee

When you stop Splunk, it has to close out all the inputs, as well as the indexes and other components of the product. The more of these you've got floating around, the longer it takes. Splunkd.log in $SPLUNK_HOME/var/log/splunk/ will tell you what is shutting down when the shutdown occurs. If your particularly curious what is taking so long, and you can't tell from splunkd.log, you can probably strace splunkd during the shutdown to see what is happening. For Windows, the equivalent tool would be procmon. I think you'll find the answer here.

View solution in original post

lguinn2
Legend

I have found that stopping Splunk takes longer when there are searches running, especially realtime searches. My guess is that Splunk sends a "stop" to each running subprocess, and then waits a bit to allow them to stop gracefully. How does that relate to your experience? Do you have many users running searches? Does Splunk stop faster if you exit the UI and do the stop command from the command line?

0 Karma

Drainy
Champion

are there any errors displayed in the splunkd.log around the time of the shutdown? Might be worth installing the SoS app and using it to have a look at your internal logs for errors or warnings (perhaps even crashes on shutdown)

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!