Deployment Architecture

Searchable rolling restart works on master node and not on another with identical configuration


We have a 8.1.2 deployment with 2 clustered indexers and a search head cluster.
We have two master nodes, one is operational, the other is on stand-by and they are configured identically.
Let's call them master1 and master2.

I want to make searchable rolling restart work in our system.

When I run splunk apply cluster-bundle on master1 when it is active, it works.
While when I make master2 the active one, I get this answer:

splunk apply cluster-bundle
Warning: Under some circumstances, this command will initiate a rolling restart of all peers. This depends on the contents of the configuration bundle.
For details, refer to the documentation. Do you wish to continue? [y/n]: y

Encountered some errors while applying the bundle.
Request rejected. Searchable rolling restart cannot be guaranteed if num of peers <= 2.

We are puzzled why it works on one master node but not on the other when they are identically configured.

Everything looks fine in the Monitoring Console whether master1 is active or master2 is.
They have the same configuration parameters for searchable_* and for rolling_restart (checked with splunk btool server list --debug)

Any suggestion about where to look for the problem?

Labels (1)
0 Karma


There is a lot more happening on a master node than just the cluster configs. What you are attempting to do is basically replace the master node with an inactive/passive node, and there are specific steps to follow. See the documentation below:


An upvote would be appreciated and Accept Solution if it helps!
0 Karma


Thank you codebuilder.

Yes, we are trying to replace the node manager and I  follow the procedure laid out in the document link you provided. The replacement worked, it was just that the two nodes did not behave exactly the same way.

In the mean time, after a number of reboots and without having changed configuration, they are now in agreement on this matter.

0 Karma
Get Updates on the Splunk Community!

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...