Deployment Architecture

"splunk restart" command takes long time

melonman
Motivator

Hi there,

When I issued "splunk restart" command, it takes more than 5 min.
Looks like stopping splunk takes most of the restart time.

Could you give us the possible reasons why restarting splunk takes longer time?

Thank you!

Tags (1)
1 Solution

jbsplunk
Splunk Employee
Splunk Employee

When you stop Splunk, it has to close out all the inputs, as well as the indexes and other components of the product. The more of these you've got floating around, the longer it takes. Splunkd.log in $SPLUNK_HOME/var/log/splunk/ will tell you what is shutting down when the shutdown occurs. If your particularly curious what is taking so long, and you can't tell from splunkd.log, you can probably strace splunkd during the shutdown to see what is happening. For Windows, the equivalent tool would be procmon. I think you'll find the answer here.

View solution in original post

jbsplunk
Splunk Employee
Splunk Employee

When you stop Splunk, it has to close out all the inputs, as well as the indexes and other components of the product. The more of these you've got floating around, the longer it takes. Splunkd.log in $SPLUNK_HOME/var/log/splunk/ will tell you what is shutting down when the shutdown occurs. If your particularly curious what is taking so long, and you can't tell from splunkd.log, you can probably strace splunkd during the shutdown to see what is happening. For Windows, the equivalent tool would be procmon. I think you'll find the answer here.

lguinn2
Legend

I have found that stopping Splunk takes longer when there are searches running, especially realtime searches. My guess is that Splunk sends a "stop" to each running subprocess, and then waits a bit to allow them to stop gracefully. How does that relate to your experience? Do you have many users running searches? Does Splunk stop faster if you exit the UI and do the stop command from the command line?

0 Karma

Drainy
Champion

are there any errors displayed in the splunkd.log around the time of the shutdown? Might be worth installing the SoS app and using it to have a look at your internal logs for errors or warnings (perhaps even crashes on shutdown)

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...