topic How to restart just a Cluster master in Deployment Architecture

How to restart just a Cluster master

hughroberts — Fri, 04 Oct 2013 22:10:13 GMT

Hello
I need to take the Cluster Master node off line to change the servers ram configuration.

Do I need to run a "splunk offline" command like for the index servers or can I just do a "splunk stop" ?

The master node is on a virtual machine so its a fairly quick re-boot operation to apply the new ram.

thanks in advance

Re: How to restart just a Cluster master

hexx — Fri, 04 Oct 2013 22:57:51 GMT

The license master can be stopped for a time without impact to the cluster peers. These will continue to index and replicate incoming data.

The cluster search-head(s) should be fine as well.

However, problems will arise if a cluster peer goes down while the master is unavailable. Do your best to avoid that.

Finally, "splunk offline" only applies to cluster peers. You do not need to use this command to stop or restart the cluster master.

Re: How to restart just a Cluster master

ChrisG — Fri, 04 Oct 2013 22:59:05 GMT

The docs have some useful information about the state your cluster will be in: http://docs.splunk.com/Documentation/Splunk/6.0/Indexer/Whathappenswhenamasternodegoesdown

Re: How to restart just a Cluster master

hexx — Sat, 05 Oct 2013 01:19:52 GMT

It should be noted that in 5.x, there is a bug (SPL-65100) that may cause the cluster to be unsearchable (The search-head will show the banner "Received an empty peer list from the master") for considerable amounts of time after the master is restarted in environments where buckets in one or more replicated indexes are aggressively frozen.

We recommend that you contact Splunk Support to learn more about the pro-active and reactive steps to take if you are experiencing this problem.

Re: How to restart just a Cluster master

davidpaper — Mon, 28 Sep 2020 15:27:16 GMT

I hit this problem too. To help the CM recover faster, have it tell the indexers to fixup buckets faster. Until all of the frozen buckets are fixed up, you'll have no functioning search. See the description here: http://docs.splunk.com/Documentation/Splunk/6.0/Indexer/Upgradeacluster#Why_the_safe_restart_cluster_master_script_is_necessary

To help recover faster, in CM's server.conf:
[clustering]
max_peer_build_load =

On my servers, I am currently running 8 in parallel and trying to get it to go higher. As soon as the bucket fixups are done, the cluster will become searchable again.