Deployment Architecture

Would it cause problems to use a Search Head Cluster with only two members?

gcusello
SplunkTrust
SplunkTrust

Hi at all,
for a customer, I need to replicate knowledge objects between two Search Heads and high availability.
The best solution is a Search Head Cluster, but the problem is that I have only two Search Heads and Splunk best practices requires at least three members.

From your experience, could I use a Search Head Cluster with only two members without great problems?

If I cannot use a Cluster, as a workaround, I thought to use a script to replicate all the knowledge object from SH1 to SH2. Can anyone else suggest a different workaround?

Bye.
Giuseppe

0 Karma
1 Solution

koshyk
Super Champion

hi Cusello, I've tried with 2 members in SHC, but was NOT successful. This mainly happens during failures, and it fails to select a captain and complains waiting for minimum members to sign-up.

It is much simpler to have a single SH and replicate configurations to another Passive SH. The trouble is, if you want to use both as active, determining which is the master-copy.

We have a setup whereby one of the SH1 is active, while SH2 is passive and we have a rsync based replication running (we created as a Splunk app and can look into how many files replicated etc.). Basically, it is an rsync -rhic option running every 5 minutes. Also we have dedicated apps for stakeholders, so all their Knowledge objects are pertained to those apps ONLY. This way we can control the rsync folders.

View solution in original post

Steve_G_
Splunk Employee
Splunk Employee

Just a clarification: A search head cluster requires a minimum of three members. It is not merely a best practice.

See http://docs.splunk.com/Documentation/Splunk/7.1.2/DistSearch/SHCsystemrequirements#Required_number_o...

gcusello
SplunkTrust
SplunkTrust

Thank you for your help, I think that this is a limitation of the Search Head Cluster and I hope that someone thinks to this!
Bye.
Giuseppe

0 Karma

koshyk
Super Champion

hi Cusello, I've tried with 2 members in SHC, but was NOT successful. This mainly happens during failures, and it fails to select a captain and complains waiting for minimum members to sign-up.

It is much simpler to have a single SH and replicate configurations to another Passive SH. The trouble is, if you want to use both as active, determining which is the master-copy.

We have a setup whereby one of the SH1 is active, while SH2 is passive and we have a rsync based replication running (we created as a Splunk app and can look into how many files replicated etc.). Basically, it is an rsync -rhic option running every 5 minutes. Also we have dedicated apps for stakeholders, so all their Knowledge objects are pertained to those apps ONLY. This way we can control the rsync folders.

gcusello
SplunkTrust
SplunkTrust

We used a script for align the second Search Head!
Thank you for your help, I think that this is a limitation of the Search Head Cluster and I hope that someone thinks to this!
Bye.
Giuseppe

0 Karma

gjanders
SplunkTrust
SplunkTrust

If you want a explanation behind why 2 node clusters are not going to work as expected refer to the consensus page of consul.io

Or refer to this Splunk page, Captain election process has deployment implications

A cluster should consist of a minimum of three members. A two-member cluster cannot tolerate any node failure. Failure of either member will prevent the cluster from electing a captain and continuing to function. Captain election requires majority (51%) assent of all members, which, in the case of a two-member cluster, means that both nodes must be running. You therefore forfeit the high availability benefits of a search head cluster if you limit it to two members.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...