Reserved bucket copies on site failure

[ptcrusher]

Suppose we're setting a multisite indexer cluster with 4 nodes in site1 and 3 nodes in site2:

[clustering]
multisite = true
available_sites = site1,site2
site_replication_factor = origin:1, total:2
site_search_factor = origin:1, total:2

What happens if we loose for instance site2 given that all sites are non-explicit sites?

According to my understanding of the documentation the cluster fix-up process will "reserve" bucket copies in site1 in preparation for the return of site2 peers given that total - explicit sites equals 2 i.e. "the search and replication factors are sufficiently large" as the documentation says: 

For non-explicit sites, the cluster reserves one searchable copy if the total components of the site's search and replication factors are sufficiently large, after handling any explicit sites, to accommodate the copy. (If the search factor isn't sufficiently large but the replication factor is, the cluster reserves one non-searchable copy.)

1. Is my understanding of the documentation correct? or i'm missing something?
2. Is there any failover timer that could be configure so the cluster fix-up process gives some room for site2 to recover before the "reserve" bucket copies start to be created?
3. Lastly should we reserve some storage in site1 to accommodate for an event where "reserve" bucket copies are created? Is there any golden number that we could use for the amount of storage that should be reserved?

Thanks in advance

[richgalloway · Accepted solution]

I received the following response to my documentation feedback  I think it clarifies the matter.

Spoiler

A "reserve copy" doesn't exist on any peer (yet).  It's as if the master is bookkeeper and the bookkeeper just makes a note in the ledger that it needs to assign that extra copy once there's a peer to assign it to.

Perhaps the docs could state something like "A 'reserve bucket copy' is a virtual copy, awaiting eventual assignment to a peer.  The copy does not actually exist yet in storage while in the reserve state. Once the master assigns it to an available peer, the bucket then gets streamed to that peer in the usual manner."

[isoutamo]

Hi
My understanding is the next. Sf & RF basically said that there is one bucket in each sites. This means that you must force manually cluster to work without site2 for normal operations.
Buckets are not copied over sites, no need for any additional space for that.
@richgalloway could you give your opinion for this?
R. Ismo

[ptcrusher]

Could you kindly gave your opinion too @richgalloway?

To be honest the documentation confused me a bit 

[richgalloway]

I don't have a good answer to this question.  I've submitted feedback on the documentation so it might be made easier to understand.  You should do the same any time you are confused by the documentation.

[ptcrusher]

Did my feedback submission too.