Deployment Architecture

How to restart just a Cluster master

hughroberts
Explorer

Hello
I need to take the Cluster Master node off line to change the servers ram configuration.

Do I need to run a "splunk offline" command like for the index servers or can I just do a "splunk stop" ?

The master node is on a virtual machine so its a fairly quick re-boot operation to apply the new ram.

thanks in advance

1 Solution

hexx
Splunk Employee
Splunk Employee

The license master can be stopped for a time without impact to the cluster peers. These will continue to index and replicate incoming data.

The cluster search-head(s) should be fine as well.

However, problems will arise if a cluster peer goes down while the master is unavailable. Do your best to avoid that.

Finally, "splunk offline" only applies to cluster peers. You do not need to use this command to stop or restart the cluster master.

View solution in original post

davidpaper
Contributor

I hit this problem too. To help the CM recover faster, have it tell the indexers to fixup buckets faster. Until all of the frozen buckets are fixed up, you'll have no functioning search. See the description here: http://docs.splunk.com/Documentation/Splunk/6.0/Indexer/Upgradeacluster#Why_the_safe_restart_cluster...

To help recover faster, in CM's server.conf:
[clustering]
max_peer_build_load =

On my servers, I am currently running 8 in parallel and trying to get it to go higher. As soon as the bucket fixups are done, the cluster will become searchable again.

hexx
Splunk Employee
Splunk Employee

The license master can be stopped for a time without impact to the cluster peers. These will continue to index and replicate incoming data.

The cluster search-head(s) should be fine as well.

However, problems will arise if a cluster peer goes down while the master is unavailable. Do your best to avoid that.

Finally, "splunk offline" only applies to cluster peers. You do not need to use this command to stop or restart the cluster master.

hexx
Splunk Employee
Splunk Employee

It should be noted that in 5.x, there is a bug (SPL-65100) that may cause the cluster to be unsearchable (The search-head will show the banner "Received an empty peer list from the master") for considerable amounts of time after the master is restarted in environments where buckets in one or more replicated indexes are aggressively frozen.

We recommend that you contact Splunk Support to learn more about the pro-active and reactive steps to take if you are experiencing this problem.

ChrisG
Splunk Employee
Splunk Employee

The docs have some useful information about the state your cluster will be in: http://docs.splunk.com/Documentation/Splunk/6.0/Indexer/Whathappenswhenamasternodegoesdown

Get Updates on the Splunk Community!

Fueling your curiosity with new Splunk ILT and eLearning courses

At Splunk Education, we’re driven by curiosity—both ours and yours! That’s why we’re committed to delivering ...

Splunk AI Assistant for SPL 1.1.0 | Now Personalized to Your Environment for Greater ...

Splunk AI Assistant for SPL has transformed how users interact with Splunk, making it easier than ever to ...

Unleash Unified Security and Observability with Splunk Cloud Platform

     Now Available on Microsoft AzureOn Demand Now Step boldly into the AI revolution with enhanced security ...