I wanted to know where the primary and secondary warm bucket copies are residing on the cluster.
For this, I ran | dbinspect
over index=_internal
for a 2 hour window and was surprised to see that, for one of the buckets, all three (replication factor is 3) buckets had name starting with rb_ whereas, generally there are 2 (n-1, if n is the RF) rb_ and 1 db_
Hard to believe it's a state transition either since the buckets seem to have not been modified in a long time (at filesystem level check)
Wondering if anyone has stumbled across the same or know if this is a genuine possibility.
Hi,
Yes this is possible, when data arrives to Indexer it create db_ bucket and n (RF -1) rb_ buckets but let's assume that Indexer which holds primary bucket went down, in this case Cluster will replicate bucket from secondary copy (Bucket starts with rb_) to another instance so new copy will be rb_ as well and in this case all 3 buckets will be rb_
It should be possible. While there are probably a number of scenarios which could cause this to happen, there's the really simple case of the indexer which held the primary no longer being available.
Since the original indexer is no longer available, cluster replication would ensure your replication factor is met. All of the buckets, however, would be replicated buckets and thus have the rb_ prefix.
Once you consider that simple case, you can imagine, there are a number of variations where you didn't lose the whole indexer, but something happened to the original bucket. I believe it is even possible where there are too many copies of a bucket and the cluster decides to remove the original copy. It may even be possible just from rebalancing the cluster.