Deployment Architecture

How to add / remove SHC members with static captain?

vgrote
Explorer

Hello,

I have a three member SHC (splunk 8.0.5.1) and want to replace the members one by one with new instances running on the same IP addresses, so first adding the new and then removing the old ones is not an option.

In my plan A, I set the SHC to static captain and then tried to remove a member on the captain's command line:

$ splunk remove shcluster-member -mgmt_uri https://1.2.3.4:8090
Raft is not initialized. This means that dynamic captain mode was not set in server.conf.

How can I remove the member without going into dynamic mode?

This has been asked before, but I saw no useful answer.

Is it possible at all? And if so how? (If not: why?)

Plan B would be juggling with ports etc but that may get a bit messy.

Thanks in advance

Volkmar

Labels (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

If you replace the members quickly enough then you may not need a static captain.  With only 3 members in the cluster, however, removing one means the SHC will not be able to elect a dynamic captain.  That means no scheduled searches will run (because the captain assigns scheduled searches to members).  If running scheduled searches is important during the maintenance period then use a static captain.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

Don't set a static captain before removing a member.  Do so after removing the member.  When the member is replaced, restore a dynamic captain and proceed with the next replacement.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

vgrote
Explorer

Hi Rich,

thanks for the swift reply.

If I do it that way, i.e. remove, static, add, dynamic, remove ..., why use a static captain at all (which will likely not accept a new member as well)?

What do I miss here?

Thanks in advance

Volkmar

0 Karma

richgalloway
SplunkTrust
SplunkTrust

If you replace the members quickly enough then you may not need a static captain.  With only 3 members in the cluster, however, removing one means the SHC will not be able to elect a dynamic captain.  That means no scheduled searches will run (because the captain assigns scheduled searches to members).  If running scheduled searches is important during the maintenance period then use a static captain.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

vgrote
Explorer

Thanks again, Rich.

That's the way I did it, lots of steps, but better be safe than sorry.

Volkmar

0 Karma

codebuilder
SplunkTrust
SplunkTrust

The captain determines whether the cluster is healthy or not by polling the other members. Be careful if you remove one of three members, for example, which can lead to a split brain situation.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!