How multisite SH clusters work

[sushree1234]

I have a multisite cluster with 3 sites . Which is having 6 indexers as peer nodes clustered across the 3-sites (2 indexers each) managed by a manager node . Also we have 2 SHs clusters across the 3-sites . 

SHcluster1 - total 9 SHs (we kept 3 SHs in each site) 
SHcluster2- total 6 SHs (we kept 2 SHs in each site) 
so wanted to understand how the configuration is going to be in deployer, each SHs ,manager node as this will be a multisite cluster .

As per my knowledge for multisite cluster - for single SHs config is
Configure the search heads
-----------------------------------
sudo ./splunk edit cluster-config -mode searchhead -site site1 -manager_uri <URI>:<mngmtPort> -secret <secretkey>

so likewise what will be the configuration for SH clusters in multisite .

[sushree1234]

we are not going to use root user for any internal config changes , its just an example i copy pasted for reference .

And my bad , i placed the number wrongly . Its 6 SHs in 1 cluster and 3 in another (those are reporting SHs) , not for normal users . 

Could you please explain the configuration . Because in splunk doc i can see the configuration for single SH . not for SH clusters with multisite indexer cluster .

[isoutamo]

You could found the explanation from here 

• https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/AboutSHC
• https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/SHCandindexercluster

If/when your SHCs have different roles then you are needing own Deployers for both, don't use the same Deployer instance if content of those SHCs are different.

It's hard to give to you any detailed explanations about your environment without more knowledge of it. And as I said, basically you have too many SHC members (SHs) vs. Indexer peers. That is something which I cannot understand with your current information.

[isoutamo]

Hi

You have a quite interesting setup! Usually the relation between SH vs IDX is something like 1:5-7. If you have very active SHC then those put your indexer cluster on it's knees!

Can you open why you have two quite big SHC with one small IDX cluster?

How to configure site settings on your SHC side? This depends on how your SHC has physically located vs. your IDX and users. Quite often there is site affinity in use when SHC nodes haven't set in specific site. Instead they are using site0 as a site information.  In that case those nodes use are IDX peer over all sites. If you have set specific site like site1 on SHC nodes in some site then those members are used only site1's indexer peers (instead of all 6) when they are doing queries.

Based on sudo ./splunk you are running your splunk as root which against the best practices and create security risk. You should change splunk run as a separate "splunk" user instead of root. See Splunk Security Guide.

r. Ismo