Deployment Architecture

"splunk restart" command takes long time

melonman
Motivator

Hi there,

When I issued "splunk restart" command, it takes more than 5 min.
Looks like stopping splunk takes most of the restart time.

Could you give us the possible reasons why restarting splunk takes longer time?

Thank you!

Tags (1)
1 Solution

jbsplunk
Splunk Employee
Splunk Employee

When you stop Splunk, it has to close out all the inputs, as well as the indexes and other components of the product. The more of these you've got floating around, the longer it takes. Splunkd.log in $SPLUNK_HOME/var/log/splunk/ will tell you what is shutting down when the shutdown occurs. If your particularly curious what is taking so long, and you can't tell from splunkd.log, you can probably strace splunkd during the shutdown to see what is happening. For Windows, the equivalent tool would be procmon. I think you'll find the answer here.

View solution in original post

jbsplunk
Splunk Employee
Splunk Employee

When you stop Splunk, it has to close out all the inputs, as well as the indexes and other components of the product. The more of these you've got floating around, the longer it takes. Splunkd.log in $SPLUNK_HOME/var/log/splunk/ will tell you what is shutting down when the shutdown occurs. If your particularly curious what is taking so long, and you can't tell from splunkd.log, you can probably strace splunkd during the shutdown to see what is happening. For Windows, the equivalent tool would be procmon. I think you'll find the answer here.

lguinn2
Legend

I have found that stopping Splunk takes longer when there are searches running, especially realtime searches. My guess is that Splunk sends a "stop" to each running subprocess, and then waits a bit to allow them to stop gracefully. How does that relate to your experience? Do you have many users running searches? Does Splunk stop faster if you exit the UI and do the stop command from the command line?

0 Karma

Drainy
Champion

are there any errors displayed in the splunkd.log around the time of the shutdown? Might be worth installing the SoS app and using it to have a look at your internal logs for errors or warnings (perhaps even crashes on shutdown)

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...