Cluster Master scalability

[danielbb]

We are in the process of virtualizing our environments and then we are facing the question of whether to use multiple cluster masters or to have fewer cluster masters that serve more indexers each. However, we don’t know how to go about it. Therefore the question, what are the scalability rules for a cluster master?

[danielbb]

Thank you all, so, we have the concept of regions, and our Splunk architecture revolves around it. So, let’s say the European one - it has the all the Splunk data of Europe in the European indexer cluster and because of that I asked the question, whether each region should have its own cluster master or they can share. If they share, how can I figure out how many buckets the cluster handles? So, we won’t reach the one million ..

[PickleRick]

There are two separate things:

One is an indexer cluster - oversimplifying a bit it's just a bunch of indexers between which the buckets might be replicated (but don't have to, I've seen clusters with RF=1; it didn't give you HA but had its pros) managed by a CM (possibly redundant in active-passive mode). The single cluster might be "stretched" across several different sites but you still need direct communication between the sites because of management traffic between CM and indexers in all sites and replication traffic between indexers themselves (again - you probably can configure multisite cluster and contain all buckets within a single site but it doesn't make much sense).

Another thing is distributed search - you can have several separate indexers or clusters and have a search head (or search head cluster) searching across all your indexers or clusters.

There is also another, even more kinky way of searching - federated search - where SH searches not directly from indexers but also from another SH. But let's leave it aside for now.

So depending on your business needs and technical constraints you might need one or another architecture.

If you have one cluster, the whole cluste has just one CM (possibly with a redundant instance). There's no "splitting cluster among several CMs". Period.

So you either need one big cluster or several smaller ones (but again - separate clusters, not one big cluster with serveral smaller CMs - there's no such thing). Which one will be appropriate in your case? That's something you should discuss with a skilled Splunk Architect - that's what you typically engage either Splunk PS or your friendly local Splunk Partner for.

[PickleRick]

Adding to @richgalloway 's answer - every cluster has exactly one active CM (even a multisite cluster). I can never recall the exact numbers but it scales to a range of millions buckets in your cluster (combined across all your indexes).

The main question is why are you asking this particular thing. What issue are you trying to resolve?

[richgalloway]

Every indexer cluster must have at least one Cluster Manager (CM).  You can opt to have one or more redundant CMs for availability.  Note that this is optional as the indexer cluster will continue to function normally if the CM is unavailable.  CMs do not scale based on the number of indexers in the cluster.

Configuring redundant CMs is not trivial.  See https://docs.splunk.com/Documentation/Splunk/9.1.2/Indexer/CMredundancy for more information.