In our setup we have a searchhead cluster with no search affinity (site0) and a multisite indexer clusters (site1/site2). Now its time for some expansion and although we already expanded the searchhead cluster it is a first for the indexer cluster. Search Tier uses the cluster master (CM) to discover the indexers. Forwarding Tier uses the indexerDiscovery i.e. also uses the cluster master (CM) to discover the indexers. The process to spawn a new indexer is pretty much automated by now and from the  https://docs.splunk.com/Documentation/Splunk/8.0.4/Indexer/Addclusterpeer it is easy to understand why a rebalance may be required. Only thing that bothers me a bit is that from the Forums there is a general guidance to putt the CM in maintenance mode (https://community.splunk.com/t5/Deployment-Architecture/Adding-a-new-indexer-to-the-indexer-cluster/td-p/298199). Any idea why it is recommended to put the CM in maintenance? Afaik the maintenance only stops the bucket fix-up operations? There's any other hidden operation that maintenance mode does? What does maintenance mode makes for a better/safer procedure? ... View more

Suppose we're setting a multisite indexer cluster with 4 nodes in site1 and 3 nodes in site2:   [clustering] multisite = true available_sites = site1,site2 site_replication_factor = origin:1, total:2 site_search_factor = origin:1, total:2     What happens if we loose for instance site2 given that all sites are non-explicit sites? According to my understanding of the documentation the cluster fix-up process will "reserve" bucket copies in site1 in preparation for the return of site2 peers given that total - explicit sites equals 2 i.e. "the search and replication factors are sufficiently large" as the documentation says:  For non-explicit sites, the cluster reserves one searchable copy if the total components of the site's search and replication factors are sufficiently large, after handling any explicit sites, to accommodate the copy. (If the search factor isn't sufficiently large but the replication factor is, the cluster reserves one non-searchable copy.)   Is my understanding of the documentation correct? or i'm missing something? Is there any failover timer that could be configure so the cluster fix-up process gives some room for site2 to recover before the "reserve" bucket copies start to be created? Lastly should we reserve some storage in site1 to accommodate for an event where "reserve" bucket copies are created? Is there any golden number that we could use for the amount of storage that should be reserved? Thanks in advance ... View more