Deployment Architecture

How to restart just a Cluster master

hughroberts
Explorer

Hello
I need to take the Cluster Master node off line to change the servers ram configuration.

Do I need to run a "splunk offline" command like for the index servers or can I just do a "splunk stop" ?

The master node is on a virtual machine so its a fairly quick re-boot operation to apply the new ram.

thanks in advance

1 Solution

hexx
Splunk Employee
Splunk Employee

The license master can be stopped for a time without impact to the cluster peers. These will continue to index and replicate incoming data.

The cluster search-head(s) should be fine as well.

However, problems will arise if a cluster peer goes down while the master is unavailable. Do your best to avoid that.

Finally, "splunk offline" only applies to cluster peers. You do not need to use this command to stop or restart the cluster master.

View solution in original post

davidpaper
Contributor

I hit this problem too. To help the CM recover faster, have it tell the indexers to fixup buckets faster. Until all of the frozen buckets are fixed up, you'll have no functioning search. See the description here: http://docs.splunk.com/Documentation/Splunk/6.0/Indexer/Upgradeacluster#Why_the_safe_restart_cluster...

To help recover faster, in CM's server.conf:
[clustering]
max_peer_build_load =

On my servers, I am currently running 8 in parallel and trying to get it to go higher. As soon as the bucket fixups are done, the cluster will become searchable again.

hexx
Splunk Employee
Splunk Employee

The license master can be stopped for a time without impact to the cluster peers. These will continue to index and replicate incoming data.

The cluster search-head(s) should be fine as well.

However, problems will arise if a cluster peer goes down while the master is unavailable. Do your best to avoid that.

Finally, "splunk offline" only applies to cluster peers. You do not need to use this command to stop or restart the cluster master.

hexx
Splunk Employee
Splunk Employee

It should be noted that in 5.x, there is a bug (SPL-65100) that may cause the cluster to be unsearchable (The search-head will show the banner "Received an empty peer list from the master") for considerable amounts of time after the master is restarted in environments where buckets in one or more replicated indexes are aggressively frozen.

We recommend that you contact Splunk Support to learn more about the pro-active and reactive steps to take if you are experiencing this problem.

ChrisG
Splunk Employee
Splunk Employee

The docs have some useful information about the state your cluster will be in: http://docs.splunk.com/Documentation/Splunk/6.0/Indexer/Whathappenswhenamasternodegoesdown

Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...