We have a 8.1.2 deployment with 2 clustered indexers and a search head cluster.
We have two master nodes, one is operational, the other is on stand-by and they are configured identically.
Let's call them master1 and master2.
I want to make searchable rolling restart work in our system.
When I run splunk apply cluster-bundle on master1 when it is active, it works.
While when I make master2 the active one, I get this answer:
splunk apply cluster-bundle
Warning: Under some circumstances, this command will initiate a rolling restart of all peers. This depends on the contents of the configuration bundle.
For details, refer to the documentation. Do you wish to continue? [y/n]: y
Encountered some errors while applying the bundle.
Request rejected. Searchable rolling restart cannot be guaranteed if num of peers <= 2.
We are puzzled why it works on one master node but not on the other when they are identically configured.
Everything looks fine in the Monitoring Console whether master1 is active or master2 is.
They have the same configuration parameters for searchable_* and for rolling_restart (checked with splunk btool server list --debug)
Any suggestion about where to look for the problem?
There is a lot more happening on a master node than just the cluster configs. What you are attempting to do is basically replace the master node with an inactive/passive node, and there are specific steps to follow. See the documentation below:
Thank you codebuilder.
Yes, we are trying to replace the node manager and I follow the procedure laid out in the document link you provided. The replacement worked, it was just that the two nodes did not behave exactly the same way.
In the mean time, after a number of reboots and without having changed configuration, they are now in agreement on this matter.