Deployment Architecture

How to restart just a Cluster master

hughroberts
Explorer

Hello
I need to take the Cluster Master node off line to change the servers ram configuration.

Do I need to run a "splunk offline" command like for the index servers or can I just do a "splunk stop" ?

The master node is on a virtual machine so its a fairly quick re-boot operation to apply the new ram.

thanks in advance

1 Solution

hexx
Splunk Employee
Splunk Employee

The license master can be stopped for a time without impact to the cluster peers. These will continue to index and replicate incoming data.

The cluster search-head(s) should be fine as well.

However, problems will arise if a cluster peer goes down while the master is unavailable. Do your best to avoid that.

Finally, "splunk offline" only applies to cluster peers. You do not need to use this command to stop or restart the cluster master.

View solution in original post

davidpaper
Contributor

I hit this problem too. To help the CM recover faster, have it tell the indexers to fixup buckets faster. Until all of the frozen buckets are fixed up, you'll have no functioning search. See the description here: http://docs.splunk.com/Documentation/Splunk/6.0/Indexer/Upgradeacluster#Why_the_safe_restart_cluster...

To help recover faster, in CM's server.conf:
[clustering]
max_peer_build_load =

On my servers, I am currently running 8 in parallel and trying to get it to go higher. As soon as the bucket fixups are done, the cluster will become searchable again.

hexx
Splunk Employee
Splunk Employee

The license master can be stopped for a time without impact to the cluster peers. These will continue to index and replicate incoming data.

The cluster search-head(s) should be fine as well.

However, problems will arise if a cluster peer goes down while the master is unavailable. Do your best to avoid that.

Finally, "splunk offline" only applies to cluster peers. You do not need to use this command to stop or restart the cluster master.

hexx
Splunk Employee
Splunk Employee

It should be noted that in 5.x, there is a bug (SPL-65100) that may cause the cluster to be unsearchable (The search-head will show the banner "Received an empty peer list from the master") for considerable amounts of time after the master is restarted in environments where buckets in one or more replicated indexes are aggressively frozen.

We recommend that you contact Splunk Support to learn more about the pro-active and reactive steps to take if you are experiencing this problem.

ChrisG
Splunk Employee
Splunk Employee

The docs have some useful information about the state your cluster will be in: http://docs.splunk.com/Documentation/Splunk/6.0/Indexer/Whathappenswhenamasternodegoesdown

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...