Deployment Architecture

What happens if during the Multisite Search Head Cluster split brain situation, we statically assign a captain on the secondary site, too?

ikulcsar
Communicator

Hi,

I have a theoretical question about Search Head Cluster(SHC) operation in a multisite environment:
- We have multiple sites.
- We have a multisite indexer cluster.
- We have an SHC cluster with nodes on both sites with Enterprise Security for example.

Situation: the two site lost the connection between them.
- The primary site can select a new captain. Life goes on on this site:)
- A captain cannot be selected dynamically on the secondary site. Just ad-hoc searches can be run.

The question: what happens if during the split brain situation we statically assign a captain on the secondary site, too (or VMware boots up all the server on both sides)? There will be 2 captains. Data accelerations, ES correlations will run and fill up indexes, summary tables, etc on the local indexer cluster on both sites.
After a recovery, can IXC merge this tables, indexer or it will throw some interesting errors? Will it accelerate, summary events twice and messed up all the notable events?

I know, the double captain is not supported, but in some environment, it can happen...

Regards,
Istvan

0 Karma

deepashri_123
Motivator

hey ikulcsar,
At no point will there be captains set twice,captain is elected internally between the search head cluster, even if all the servers are boot up, the captain will be elected between the cluster members , so there will be no double captaincy.

0 Karma

ikulcsar
Communicator

Hey,

Thank you for your fast response.

I can understand in normal operations condition, there cannot be two captains. But, when the two site separated totally because a network failer, AND on the smaller site (with less than half of the SH nodes), a captain can be delegated manually. I didn't that it is a rational action, but it can be done.

regards
Istvan

0 Karma
*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!

Review:





Or Learn More in Our Blog >>