Is there a reason the minimum number of nodes for indexer clustering needs to be 3?

[munang]

Is there a reason the minimum number of nodes for indexer clustering needs to be 3?

If three units are needed because of the role of parity in the raid theory, I don't think this role is necessary because the CM is already doing it. Therefore, I think that 2 units should also play a clustering role, but I wonder why 3 units always come out as default in most examples. Is there any other reason??

[munang]

@gcusello 

thanks for the answer

I wasn't confused with search header clustering, but I was also curious about the reason because the maximum number of search header clustering is 3.

In my opinion, not only in search header clustering, but also in other big data systems, I found out that there is a majority principle when configuring search nodes, and at this time, it is more efficient to configure clustering with odd numbers rather than even numbers for majority voting.

Comparatively fewer nodes are required as the majority requires 3 units when configuring 4 units and the majority requires 3 units when configuring 5 units. Therefore, it is configured with an odd number, and at this time, the minimum number of clustering is 3, so I think the minimum number is 3.

[PickleRick]

Again, close but no banana 😉

There is a minimum of 3 search heads for a search heads cluster because otherwise the voting won't work. You can force two-node search head cluster to work with a manually designated captain but it has its limitations. There are no other "formal requirements" meaning that there is no requirement that number of search heads must be odd or even or such but there are factors to take into account when architecting more complicated setups, especially spread across different DCs when you can lose significant part of your servers. But that's an advanced topic 😉

And again - there is no minimum required servers for indexer cluster. In fact when you're configuring a cluster you often start with a CM and connect indexers one by one so you have a single-indexer cluster for a short time. Of course a cluster without replication doesn't protect you against data loss but it's still a cluster.

I agree that typically when you talk about a cluster you mean a RF>=2 cluster but it's not a technical necessity. You can even make a single indexer multisite cluster. So technically you don't have any minimum here.

Best pratcice, however, is another thing. If you have RF=2 (the typical case), you want to have at least three nodes because with just two nodes if one goes down you don't have any spare servers to replicate your data to. Think RAID with/without hot-spare.

[gcusello]

Hi @munang,

in SHC, three is the minumum number of SHs not the maximum, you can escalate how much you need.

Your other considerations aren't applicable to Splunk: I don't know your experience, but in Splunk there isn't any rule about odd or pair number of Indexers or Search Heads:

• for Search Head Cluster you need at least 3 SHs and you choose the number based on the scheduled searches and the consurrent users,
• for Indexers Cluster, as I said, you need at least two indexers for HA and you choose the number of Indexers based on the work that they have to do: data to index, concurrent users, scheduled searches, Premium Apps.

e.g., if you have to index 800 GB/day and a normal number of concurrent users and scheduled searches, you should have 4 Indexers, not 3 or 5.

I hint to follow the Splunk Architect certification path, for architecture design and system dimensioning.

Ciao.

Giuseppe

[jotne]

i did read (slak) that you do not need to have two copies of indexes to use index cluster.  It could give you benefit when adding more nodes to the index since it then distribute data over the new index nodes better.

There may also be other benefit as well with management of the index nodes etc.

PS I have not tested this

[PickleRick]

There is no such requirement. You can even have a relatively healthy (albeit completely useless from the data resilience point of view) cluster with just one indexer. Of course you'd need to have RF=1 for such setup so the buckets would not try to replicate but however strange it might seem it actually makes sense.

If you add a standalone indexer to the cluster, the buckets that were indexed before remain unclustered and will not get replicated. If you have a "one node cluster" and then add another peer(s) to it, if you raise your RF, the buckets you have are already clustered and will get replicated. Also if you want to add more nodes and still _not_ set up any replication with such "degraded" cluster with RF=1 you have the possibility to rebalance buckets across your nodes.

What you're referring to as the minimal size might be the size requirement for search head cluster. It needs to be at least 3 nodes big so that automatic captain election can happen.

[gcusello · Accepted solution]

Hi @munang,

The minimum number of peers in an Indexer Cluster is two, having at least two copies od data, plus the Master Node to manage the Cluster peers and the replications between them.
The number of peers is determined by

• the volume of data to acquire,
• the number of concurrent users,
• the number of active Scheduled searches,
• the presence of Premium Apps such as Enterprise Security or ITSI;

an indicative value can start from the value of an Indexer every 200 GB/day to acquire, if you don't have ES or ITSI, and obviously they must be at least two.
Perhaps you are confusing the Indexer Cluster with the Search Head Cluster, where (I can't tell you why) at least three Search Heads plus the Deployer are required for the Cluster.

Ciao.

Giuseppe