Deployment Architecture

How to restart just a Cluster master

hughroberts
Explorer

Hello
I need to take the Cluster Master node off line to change the servers ram configuration.

Do I need to run a "splunk offline" command like for the index servers or can I just do a "splunk stop" ?

The master node is on a virtual machine so its a fairly quick re-boot operation to apply the new ram.

thanks in advance

1 Solution

hexx
Splunk Employee
Splunk Employee

The license master can be stopped for a time without impact to the cluster peers. These will continue to index and replicate incoming data.

The cluster search-head(s) should be fine as well.

However, problems will arise if a cluster peer goes down while the master is unavailable. Do your best to avoid that.

Finally, "splunk offline" only applies to cluster peers. You do not need to use this command to stop or restart the cluster master.

View solution in original post

davidpaper
Contributor

I hit this problem too. To help the CM recover faster, have it tell the indexers to fixup buckets faster. Until all of the frozen buckets are fixed up, you'll have no functioning search. See the description here: http://docs.splunk.com/Documentation/Splunk/6.0/Indexer/Upgradeacluster#Why_the_safe_restart_cluster...

To help recover faster, in CM's server.conf:
[clustering]
max_peer_build_load =

On my servers, I am currently running 8 in parallel and trying to get it to go higher. As soon as the bucket fixups are done, the cluster will become searchable again.

hexx
Splunk Employee
Splunk Employee

The license master can be stopped for a time without impact to the cluster peers. These will continue to index and replicate incoming data.

The cluster search-head(s) should be fine as well.

However, problems will arise if a cluster peer goes down while the master is unavailable. Do your best to avoid that.

Finally, "splunk offline" only applies to cluster peers. You do not need to use this command to stop or restart the cluster master.

hexx
Splunk Employee
Splunk Employee

It should be noted that in 5.x, there is a bug (SPL-65100) that may cause the cluster to be unsearchable (The search-head will show the banner "Received an empty peer list from the master") for considerable amounts of time after the master is restarted in environments where buckets in one or more replicated indexes are aggressively frozen.

We recommend that you contact Splunk Support to learn more about the pro-active and reactive steps to take if you are experiencing this problem.

ChrisG
Splunk Employee
Splunk Employee

The docs have some useful information about the state your cluster will be in: http://docs.splunk.com/Documentation/Splunk/6.0/Indexer/Whathappenswhenamasternodegoesdown

Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...