Deployment Architecture

Searchable rolling restart works on master node and not on another with identical configuration

Marlan
Explorer

We have a 8.1.2 deployment with 2 clustered indexers and a search head cluster.
We have two master nodes, one is operational, the other is on stand-by and they are configured identically.
Let's call them master1 and master2.

I want to make searchable rolling restart work in our system.

When I run splunk apply cluster-bundle on master1 when it is active, it works.
While when I make master2 the active one, I get this answer:

splunk apply cluster-bundle
Warning: Under some circumstances, this command will initiate a rolling restart of all peers. This depends on the contents of the configuration bundle.
For details, refer to the documentation. Do you wish to continue? [y/n]: y

Encountered some errors while applying the bundle.
Request rejected. Searchable rolling restart cannot be guaranteed if num of peers <= 2.


We are puzzled why it works on one master node but not on the other when they are identically configured.

Everything looks fine in the Monitoring Console whether master1 is active or master2 is.
They have the same configuration parameters for searchable_* and for rolling_restart (checked with splunk btool server list --debug)

Any suggestion about where to look for the problem?

Labels (1)
0 Karma

codebuilder
SplunkTrust
SplunkTrust

There is a lot more happening on a master node than just the cluster configs. What you are attempting to do is basically replace the master node with an inactive/passive node, and there are specific steps to follow. See the documentation below:

https://docs.splunk.com/Documentation/Splunk/8.2.1/Indexer/Handlemanagernodefailure

 

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

Marlan
Explorer

Thank you codebuilder.

Yes, we are trying to replace the node manager and I  follow the procedure laid out in the document link you provided. The replacement worked, it was just that the two nodes did not behave exactly the same way.

In the mean time, after a number of reboots and without having changed configuration, they are now in agreement on this matter.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...