How to scale out a multisite cluster

[chenfan]

Hi Splunkers,

I recently encountered a requirement to expand our existing multisite indexer cluster. We need to add new sites and deploy search heads in the new sites. These search heads must join the existing search head cluster, but each site’s search heads should only search data stored locally in their own site.

Current Cluster Manager (Master) Configuration

[general]
site = site0

[clustering]
available_sites = site5,site6,site7,site8
mode = master
multisite = true
pass4SymmKey = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
site_replication_factor = origin:1,site8:1,total:2
site_search_factor = origin:1,site8:1,total:2
rebalance_threshold = 0.9
cluster_label = eg_idxc
maintenance_mode = false
site_mappings = site1:site8,site2:site6,site3:site7,site4:site5
constrain_singlesite_buckets = false
replication_factor = 2

Planned Changes

We plan to add site9, site10, site11, and site12. Site12 must also store a copy of the data.

The updated Cluster Manager configuration​ will be:

[general]
site = site0

[clustering]
available_sites = site5,site6,site7,site8,site9,site10,site11,site12
mode = master
multisite = true
site_replication_factor = origin:1,site8:1,site12:1,total:3
site_search_factor = origin:1,site8:1,site12:1,total:3
rebalance_threshold = 0.9
cluster_label = eg_idxc
maintenance_mode = false
site_mappings = site1:site8,site2:site6,site3:site7,site4:site5
constrain_singlesite_buckets = false
replication_factor = 3

Indexer Expansion (Example: site12)

Add indexer peers to the multisite cluster:
splunk edit cluster-config -mode peer -site site1 -manager_uri https://$CM_IP$:8089 -replication_port 9887 -secret $cluster_secret_password$

Search Head Expansion (New Site Search Heads)

Initialize Search Head Cluster Config：
splunk init shcluster-config -auth <username>:<password> -mgmt_uri <URI>:<management_port> -replication_port <replication_port> -replication_factor <n> -conf_deploy_fetch_url <URL>:<management_port> -secret <security_key> -shcluster_label <label>

Join the Existing SHC and Bind to site12：
splunk edit cluster-config -mode searchhead -site site12 -master_uri https://$CM_IP$:8089 -secret $cluster_secret_password$ -auth login:password

If anyone has suggestions or sees potential issues with this approach—especially around site affinity behavior​ or search factor design—please let me know.

Thanks in advance!

[isoutamo]

As other already said there is more issues than you want to handle with your requirements.

Can you open more about your current setup, what you are really try to achieve with this new setup and where these sites are in geographically point of view. 

I didn’t understand your configurations and requirements based what you have told to us. We definitely need more information where you are put nodes and how those are located currently. E.g. you have site_search_factor 2 and you want that SHC nodes are searching only on local site. You don’t tell if your current SHC node is only on site8 or not? Based on your requirements this she that way. Then you want add additional SHC nodes into site12. 

Why you have indexers on 4 in future 8 site, but use only two sites data for searching? 

How many indexers you have per site and how many SHCs you totally have and in which sites?

Why you want to put new indexers into old removed site1?

Why you have that site definition in general?

[PickleRick]

Adding a bit to @livehybrid 's answer - SHCs are meant as way of scaling performance, not providing HA. "Some level" of HA is a side effect.

So.

1. Whole SHC must be either in one site or in site0. There is no way of splitting single SHC across multiple logical sites, let alone forcing/banning artifact replication across those sites.

2. Splitting your cluster across physical sites can have effect on your experience due to possible higher latency but also can leave you in a split-brain scenarios or with only a minority part of your cluster in case of a site outage. Some of those scenarios can be remediated but it's not straightforward and may require quite a lot of manual work.

I suspect the underlying idea is to create a SHC so that the configurations are propagated across the whole cluster but the searching is done locally within a single site. Unless you have users often "migrating" across those sites and using their own knowledge objects that's something which should rather be done by proper configuration management process, probably with centralized version control system and/or deployment server.

[livehybrid]

Hi @chenfan 

By having a single replica copy one each site you might find that there is a lot of cross-site replication when there are issues, typically this would take longer than replicating a bucket from the same site. 

Regarding 'each site’s search heads should only search data stored locally in their own site' - with site affinity it will search the site *if possible* which means it will get the buckets from another site if it has to (see https://help.splunk.com/en/splunk-enterprise/administer/manage-indexers-and-indexer-clusters/10.4/de....)

Having a single SHC across multiple sites worries me a little - there can be alot of chatter/traffic between the SHC members to keep them in sync through replication/election etc. Splunk guidance is that SHC members need low-latency links between all members. 

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful

• Marking it as the solution if it resolved your issue

• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing.