question related to search head captain

[jatin3101]

Hello.

I have some question about the captain selection process.

(i am very new to splunk its only been 2 months so if these things are obvious answer them regardless if possible also attach the link of the official documentation so i can verify them )

q1.lets say i have a cluster of 5 members & if the captain goes down -- so their are remaining 4 left how can the election run in an even number of member how will the voting happen 

in my opinion their will be a random timer and the ones whose timer if finished first ask everyone to vote him
and usually they do vote for him.

q2.if the election was successful in first case why it is advised to have odd number of members in cluster why cant from beginning i have a cluster of 6 members ?

q3.what if the voting done like this in case of 5 member cluster (1 vote to 2 /2 to 3/3 to 4/4 to 5/5to 1)  
is it even possible if no then why ?
 

[asimit]

Hi @jatin3101 , no worries being new to this - SHC captain stuff can be tricky at first. I'll try to break it down simple for your questions what I know so far. Check the official docs too for the full deets.

Q1: Election with 4 members left (from 5)
Yeah, your guess on the random timer is spot on. All members have a random election timer (like election_timeout_ms, defaults to something around 60s I think). The one that finishes first says "hey vote for me" to everyone - including the dead captain's spot. Needs majority of total members, so for 5 total, thats 3 votes. The 4 alive ones can give those 3 easy, even if captain is down. It worked like that in my 5-node cluster when we lost one.

https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/9.2/configuration-file-reference/9.2.11-configuration-file-reference/server.conf#:~:text=members.%0A*%20Default%3A%20false-,election_timeout_ms,-%3D%20%3Cpositive_integer%3E%0A*%20The%20amount

election_timeout_ms = <positive_integer>
* The amount of time, in milliseconds, that a member waits before
trying to become the captain.
* Note that modifying this value can alter the heartbeat period (See
election_timeout_2_hb_ratio for further details)
* A very low value of election_timeout_ms can lead to unnecessary captain
elections.
* Default: 60000 (1 minute)

Q2: Why odd number recommended over even like 6?
Majority is always (N/2)+1 of total members, no matter how many alive. For 6, needs 4 votes. If 3 go down, you're stuck at 3 alive - no captain possible till more recover. With odd like 5 (needs 3), losing 2 leaves 3 alive, still elects fine. Even numbers risk split quorums easier, like 3-3. Docs push 3,5,7 etc for that reason. Not that 6 never works, but riskier.​

Q3: That circular voting thing
Nah, not how it works at all. It's not pre-assigned votes like 1->2 etc. Candidate asks all members (alive ones respond), they vote yes/no based on rules (like preferred_captain, in-sync status). First timer proposer usually wins if quorum ok, cuz others agree quick. No cycle voting possible.
​

Hope that clears it up! Heres the main doc link:
Splunk Official Doc:
https://help.splunk.com/en/splunk-enterprise/administer/distributed-search/9.2/manage-search-head-cl...

Old article from HurricaneLabs but good one:
https://hurricanelabs.com/splunk-tutorials/the-myth-of-the-three-member-search-head-cluster/

Please give karma 👍 for support 😁 happly splunking .... 😎

[isoutamo]

On more comment for this.
When you have multisite configurations with SHC nodes divided to several sites. Then those election requirements are easier to fulfill when one site crashed.

[jatin3101]

see i understand the 1 and 3 question 

i am just asking if the election was possible in the 4 sh member why cant i have even members to begin with lets say i have and example 

we are about to face 3 cluster member failure(SH)

case 1- we have a 7 member cluster -- 3 fails , 4 are still left new captain will be elected by voting 
case 2 - we have 8 member cluster -- 3 fails ,5 are still left the new captain can still be elected 

so why is case 1 acceptable and case 2 is not  

[PickleRick]

Splunk Search Head clusters use RAFT algorithm to keep the state of the cluster - https://en.wikipedia.org/wiki/Raft_(algorithm)

If you want to dig deeper into that - there are quite a lot of materials about it on the internet.

Anyway, as others already pointed out, to choose a captain, a quorum of floor(N/2)+1 members is needed. So in a split-brain scenario with even number of member split in the middle, there is no way of choosing the captain.

That's the "most common" part of the answer.

But there are other factors. Typically if we're talking about the number of member nodes and possible split brain scenario there is a silent assumption that those nodes are somehow (usually evenly) distributed into more than one site - that's when the split brain scenario is most probable.

The less common part of the answer is that there are some ways of dealing with this limitation:

1) You can designate the captain manually.

2) You can have a "voting only" SHC member - a small machine not being an "active" part of a cluster (not getting any searches to run) making sure that you have a quorum.

The general rule of thumb that you should have odd number of nodes in your SHC is just so that the cluster can handle whole site outages without any need for manual intervention (which still might not be 100% true because if you spread your users' traffic to the nodes only in local site, you still might end up with some issues depending on how your LBs are configured).

[gcusello]

Hi @jatin3101 ,

they are both acceptable:

you need as available at least more than 51% of all peers, so if you have 8 peers, you can support 3 peers down.

Anyway, usually SHs are odd and not pair.

Ciao.

Giuseppe

[gcusello]

Hi @jatin3101 ,

in general, for more infos see at https://help.splunk.com/en/splunk-enterprise/administer/distributed-search/9.4/overview-of-search-he... 

anyway, trying to answer to your questions:

q1:

if the Captaing goes down, you still have 4 SHs, so you have more than 51% of all peers (not active peers) that can elect a new Captain, there could be an issue if the Captain and other two peers goes down because you will not have 51% of the peers active.

q2:

You need an odd number of members (like 3, 5, 7) in a Splunk Search Head Cluster (SHC) to guarantee a majority vote during captain elections, preventing split-brain scenarios; a 6-member cluster (even) requires 4 votes for a majority, but if 3 members fail, the remaining 3 can't reach that 4-vote threshold, causing the cluster to fail, whereas with 5 members (requiring 3 votes), losing 2 still leaves 3 members to elect a new captain, ensuring continuous operation. 

q3:

the Captain election is random, so it is statistically impossible to have the condition you described.

The election process involves timers set randomly on all the members. The member whose timer runs out first stands for election and asks the other members to vote for it. Usually, the other members comply and that member becomes the new captain.

Ciao.

Giuseppe