Deployment Architecture

How to restart a SHC (search head cluster) at the server level?

Jason
Motivator

I am at a client where, by policy, they must restart servers every week. They have an 8-node Search Head Cluster. What is the best method for restarting it?

(Is there a maintenance mode, such as indexer clustering? Do they need to run any command before/after the restart? Should they restart 1, 2, 3, 8 at a time?)

0 Karma
1 Solution

mahamed_splunk
Splunk Employee
Splunk Employee

There is no maintenance mode in SHC. The nodes can be restarted in any order you want. It's a question of whether you want to maintain availability during the restart process. If availability is not required, then you can restart them all at once.

View solution in original post

pkumar9610
Explorer

To perform rolling restart of SH cluster use,

splunk rolling-restart shcluster-members

To check the current status of rolling restart use, 

splunk rolling-restart shcluster-members -status 1

sowings
Splunk Employee
Splunk Employee
0 Karma

Jason
Motivator

I need to know how to restart the servers themselves, not run a rolling restart on the Splunk instances.

0 Karma

IamaRobot
New Member

This is a link to the docs, but the docs don't address this question. We want to know if rolling restarts perform what I would call a "graceful" restart. For a good description of how a graceful restart should work see this description from Apache https://httpd.apache.org/docs/2.4/stopping.html#graceful

Users would expect a graceful restart to dis allow new searches, but allow currently running searches to finish before restarting.

0 Karma

IamaRobot
New Member

This should have appeared as a reply to "sowings".

0 Karma

sowings
Splunk Employee
Splunk Employee

Splunk won't trigger a restart of the host OS. Maintenance mode is not required, because the SHC is a bit less paranoid about satisfying replication of the artifacts. We're not talking about data fidelity, we're talking about cached copies of the searches that have been run. If you're talking about replication of knowledge objects, that will always happen across all nodes.

0 Karma

Jason
Motivator

Yes. I'm saying a restart of the host OS is required by policy, and I needed to know the best way to do it for the clustered search heads. It sounds like all at once is sufficient.

0 Karma

mahamed_splunk
Splunk Employee
Splunk Employee

There is no maintenance mode in SHC. The nodes can be restarted in any order you want. It's a question of whether you want to maintain availability during the restart process. If availability is not required, then you can restart them all at once.

Jason
Motivator

Great - thanks. Restarting them all servers at once will not cause unnecessary replication, assuming that some may come back online before others?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...