Deployment Architecture

Searchable rolling restart works on master node and not on another with identical configuration

Marlan
Explorer

We have a 8.1.2 deployment with 2 clustered indexers and a search head cluster.
We have two master nodes, one is operational, the other is on stand-by and they are configured identically.
Let's call them master1 and master2.

I want to make searchable rolling restart work in our system.

When I run splunk apply cluster-bundle on master1 when it is active, it works.
While when I make master2 the active one, I get this answer:

splunk apply cluster-bundle
Warning: Under some circumstances, this command will initiate a rolling restart of all peers. This depends on the contents of the configuration bundle.
For details, refer to the documentation. Do you wish to continue? [y/n]: y

Encountered some errors while applying the bundle.
Request rejected. Searchable rolling restart cannot be guaranteed if num of peers <= 2.


We are puzzled why it works on one master node but not on the other when they are identically configured.

Everything looks fine in the Monitoring Console whether master1 is active or master2 is.
They have the same configuration parameters for searchable_* and for rolling_restart (checked with splunk btool server list --debug)

Any suggestion about where to look for the problem?

Labels (1)
0 Karma

codebuilder
SplunkTrust
SplunkTrust

There is a lot more happening on a master node than just the cluster configs. What you are attempting to do is basically replace the master node with an inactive/passive node, and there are specific steps to follow. See the documentation below:

https://docs.splunk.com/Documentation/Splunk/8.2.1/Indexer/Handlemanagernodefailure

 

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

Marlan
Explorer

Thank you codebuilder.

Yes, we are trying to replace the node manager and I  follow the procedure laid out in the document link you provided. The replacement worked, it was just that the two nodes did not behave exactly the same way.

In the mean time, after a number of reboots and without having changed configuration, they are now in agreement on this matter.

0 Karma
Get Updates on the Splunk Community!

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...