Deployment Architecture

How to add / remove SHC members with static captain?

vgrote
Path Finder

Hello,

I have a three member SHC (splunk 8.0.5.1) and want to replace the members one by one with new instances running on the same IP addresses, so first adding the new and then removing the old ones is not an option.

In my plan A, I set the SHC to static captain and then tried to remove a member on the captain's command line:

$ splunk remove shcluster-member -mgmt_uri https://1.2.3.4:8090
Raft is not initialized. This means that dynamic captain mode was not set in server.conf.

How can I remove the member without going into dynamic mode?

This has been asked before, but I saw no useful answer.

Is it possible at all? And if so how? (If not: why?)

Plan B would be juggling with ports etc but that may get a bit messy.

Thanks in advance

Volkmar

Labels (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

If you replace the members quickly enough then you may not need a static captain.  With only 3 members in the cluster, however, removing one means the SHC will not be able to elect a dynamic captain.  That means no scheduled searches will run (because the captain assigns scheduled searches to members).  If running scheduled searches is important during the maintenance period then use a static captain.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

Don't set a static captain before removing a member.  Do so after removing the member.  When the member is replaced, restore a dynamic captain and proceed with the next replacement.

---
If this reply helps you, Karma would be appreciated.
0 Karma

vgrote
Path Finder

Hi Rich,

thanks for the swift reply.

If I do it that way, i.e. remove, static, add, dynamic, remove ..., why use a static captain at all (which will likely not accept a new member as well)?

What do I miss here?

Thanks in advance

Volkmar

0 Karma

richgalloway
SplunkTrust
SplunkTrust

If you replace the members quickly enough then you may not need a static captain.  With only 3 members in the cluster, however, removing one means the SHC will not be able to elect a dynamic captain.  That means no scheduled searches will run (because the captain assigns scheduled searches to members).  If running scheduled searches is important during the maintenance period then use a static captain.

---
If this reply helps you, Karma would be appreciated.

vgrote
Path Finder

Thanks again, Rich.

That's the way I did it, lots of steps, but better be safe than sorry.

Volkmar

0 Karma

codebuilder
SplunkTrust
SplunkTrust

The captain determines whether the cluster is healthy or not by polling the other members. Be careful if you remove one of three members, for example, which can lead to a split brain situation.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma
Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

REGISTER NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more ...

Security Highlights | November 2022 Newsletter

 November 2022 2022 Gartner Magic Quadrant for SIEM: Splunk Named a Leader for the 9th Year in a RowSplunk is ...

Platform Highlights | November 2022 Newsletter

 November 2022 Skill Up on Splunk with our New Builder Tech Talk SeriesCan you build it? Yes you can! *play ...