Why are we unable to remove excess buckets in a multisite indexer clustering environment?

[rbal_splunk]

We are in Multi-site clustered environment with Site RF and SF= 3 and for different indexes, we see a number of buckets listed as excess buckets. Due to some reason, we are unable to remove these buckets

[rbal_splunk · Accepted solution]

Run the search below to get the list of extra buckets - you will run this on the cluster master. Note I have set “replication_count>3” which means it will list only buckets that have a number of copies above 3.

| rest /services/cluster/master/buckets filter=replication_count>3| rex field=title "^(?<repl_index>[^\~]+)"
   | search repl_index="*" standalone=0 frozen=0
   | rename title AS bucketID
   | fields bucketID peers.*.search_state  *site*
   | untable bucketID siteState value
   | rex field=siteState "peers\.(?<peerGUID>[^\.]*?)\.(?<siteState>search_state)"
   | rex field=siteState "(?<siteState>primaries_by_site)\.(?<site>\S+)"
   | rex field=siteState "(?<siteState>rep_count_by_site)\.(?<site>\S+)"
   | rex field=siteState "(?<siteState>search_count_by_site)\.(?<site>\S+)"
   | eval peerGUID=if(siteState=="primaries_by_site", value, peerGUID)
   | eval site=if(siteState=="origin_site", value, site)
   | eval value=if(siteState=="search_count_by_site", site + ":" + value, value)
   | eval value=if(siteState=="rep_count_by_site", site + ":" + value, value)
   | join type=outer peerGUID [ rest /services/cluster/master/peers
                          | fields active_* host* label title status site
                          | eval PeerName= site + ":" + label + ":" + host_port_pair
                          | rename title AS peerGUID
                          | rename site AS peerSite
                          | table peerGUID PeerName peerSite ]
   | eval site=if(siteState=="search_state", peerSite, site)
   | eval value=if(siteState=="primaries_by_site", PeerName + ":For_" + site, value)
   | eval value=if(siteState=="search_state", PeerName + ":" + value, value)
   | fields - PeerName peerGUID peerSite    | chart values(value) over bucketID by siteState 

For the list of the Buckets listed - check the Bucket REST end point using URL

https://:/services/cluster/master/buckets/

like.....

https://:/services/cluster/master/buckets/_audit~1~ABE7B836-1BD4-4EBD-8F2F-740DAE1DB9F4

check these buckets' REST end point for attribute "constrain_to_origin_site" . If this attribute has a value of '1' - it means that bucket was created before enabling multisite. These buckets won't be removed for excess copies.