Deployment Architecture

Why are we unable to remove excess buckets in a multisite indexer clustering environment?

rbal_splunk
Splunk Employee
Splunk Employee

We are in Multi-site clustered environment with Site RF and SF= 3 and for different indexes, we see a number of buckets listed as excess buckets. Due to some reason, we are unable to remove these buckets

0 Karma
1 Solution

rbal_splunk
Splunk Employee
Splunk Employee

Run the search below to get the list of extra buckets - you will run this on the cluster master. Note I have set “replication_count>3” which means it will list only buckets that have a number of copies above 3.

| rest /services/cluster/master/buckets filter=replication_count>3| rex field=title "^(?<repl_index>[^\~]+)"
   | search repl_index="*" standalone=0 frozen=0
   | rename title AS bucketID
   | fields bucketID peers.*.search_state  *site*
   | untable bucketID siteState value
   | rex field=siteState "peers\.(?<peerGUID>[^\.]*?)\.(?<siteState>search_state)"
   | rex field=siteState "(?<siteState>primaries_by_site)\.(?<site>\S+)"
   | rex field=siteState "(?<siteState>rep_count_by_site)\.(?<site>\S+)"
   | rex field=siteState "(?<siteState>search_count_by_site)\.(?<site>\S+)"
   | eval peerGUID=if(siteState=="primaries_by_site", value, peerGUID)
   | eval site=if(siteState=="origin_site", value, site)
   | eval value=if(siteState=="search_count_by_site", site + ":" + value, value)
   | eval value=if(siteState=="rep_count_by_site", site + ":" + value, value)
   | join type=outer peerGUID [ rest /services/cluster/master/peers
                          | fields active_* host* label title status site
                          | eval PeerName= site + ":" + label + ":" + host_port_pair
                          | rename title AS peerGUID
                          | rename site AS peerSite
                          | table peerGUID PeerName peerSite ]
   | eval site=if(siteState=="search_state", peerSite, site)
   | eval value=if(siteState=="primaries_by_site", PeerName + ":For_" + site, value)
   | eval value=if(siteState=="search_state", PeerName + ":" + value, value)
   | fields - PeerName peerGUID peerSite    | chart values(value) over bucketID by siteState 

For the list of the Buckets listed - check the Bucket REST end point using URL

https://:/services/cluster/master/buckets/

like.....

https://:/services/cluster/master/buckets/_audit~1~ABE7B836-1BD4-4EBD-8F2F-740DAE1DB9F4

check these buckets' REST end point for attribute "constrain_to_origin_site" . If this attribute has a value of '1' - it means that bucket was created before enabling multisite. These buckets won't be removed for excess copies.

View solution in original post

rbal_splunk
Splunk Employee
Splunk Employee

Run the search below to get the list of extra buckets - you will run this on the cluster master. Note I have set “replication_count>3” which means it will list only buckets that have a number of copies above 3.

| rest /services/cluster/master/buckets filter=replication_count>3| rex field=title "^(?<repl_index>[^\~]+)"
   | search repl_index="*" standalone=0 frozen=0
   | rename title AS bucketID
   | fields bucketID peers.*.search_state  *site*
   | untable bucketID siteState value
   | rex field=siteState "peers\.(?<peerGUID>[^\.]*?)\.(?<siteState>search_state)"
   | rex field=siteState "(?<siteState>primaries_by_site)\.(?<site>\S+)"
   | rex field=siteState "(?<siteState>rep_count_by_site)\.(?<site>\S+)"
   | rex field=siteState "(?<siteState>search_count_by_site)\.(?<site>\S+)"
   | eval peerGUID=if(siteState=="primaries_by_site", value, peerGUID)
   | eval site=if(siteState=="origin_site", value, site)
   | eval value=if(siteState=="search_count_by_site", site + ":" + value, value)
   | eval value=if(siteState=="rep_count_by_site", site + ":" + value, value)
   | join type=outer peerGUID [ rest /services/cluster/master/peers
                          | fields active_* host* label title status site
                          | eval PeerName= site + ":" + label + ":" + host_port_pair
                          | rename title AS peerGUID
                          | rename site AS peerSite
                          | table peerGUID PeerName peerSite ]
   | eval site=if(siteState=="search_state", peerSite, site)
   | eval value=if(siteState=="primaries_by_site", PeerName + ":For_" + site, value)
   | eval value=if(siteState=="search_state", PeerName + ":" + value, value)
   | fields - PeerName peerGUID peerSite    | chart values(value) over bucketID by siteState 

For the list of the Buckets listed - check the Bucket REST end point using URL

https://:/services/cluster/master/buckets/

like.....

https://:/services/cluster/master/buckets/_audit~1~ABE7B836-1BD4-4EBD-8F2F-740DAE1DB9F4

check these buckets' REST end point for attribute "constrain_to_origin_site" . If this attribute has a value of '1' - it means that bucket was created before enabling multisite. These buckets won't be removed for excess copies.

View solution in original post

.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!