How to add / remove SHC members with static captain?

[vgrote]

Hello,

I have a three member SHC (splunk 8.0.5.1) and want to replace the members one by one with new instances running on the same IP addresses, so first adding the new and then removing the old ones is not an option.

In my plan A, I set the SHC to static captain and then tried to remove a member on the captain's command line:

$ splunk remove shcluster-member -mgmt_uri https://1.2.3.4:8090
Raft is not initialized. This means that dynamic captain mode was not set in server.conf.

How can I remove the member without going into dynamic mode?

This has been asked before, but I saw no useful answer.

Is it possible at all? And if so how? (If not: why?)

Plan B would be juggling with ports etc but that may get a bit messy.

Thanks in advance

Volkmar

[richgalloway]

Don't set a static captain before removing a member.  Do so after removing the member.  When the member is replaced, restore a dynamic captain and proceed with the next replacement.

[vgrote]

Hi Rich,

thanks for the swift reply.

If I do it that way, i.e. remove, static, add, dynamic, remove ..., why use a static captain at all (which will likely not accept a new member as well)?

What do I miss here?

Thanks in advance

Volkmar

[richgalloway · Accepted solution]

If you replace the members quickly enough then you may not need a static captain.  With only 3 members in the cluster, however, removing one means the SHC will not be able to elect a dynamic captain.  That means no scheduled searches will run (because the captain assigns scheduled searches to members).  If running scheduled searches is important during the maintenance period then use a static captain.

[vgrote]

Thanks again, Rich.

That's the way I did it, lots of steps, but better be safe than sorry.

Volkmar

[codebuilder]

The captain determines whether the cluster is healthy or not by polling the other members. Be careful if you remove one of three members, for example, which can lead to a split brain situation.