Deployment Architecture

Why are we unable to remove excess buckets in a multisite indexer clustering environment?

rbal_splunk
Splunk Employee
Splunk Employee

We are in Multi-site clustered environment with Site RF and SF= 3 and for different indexes, we see a number of buckets listed as excess buckets. Due to some reason, we are unable to remove these buckets

0 Karma
1 Solution

rbal_splunk
Splunk Employee
Splunk Employee

Run the search below to get the list of extra buckets - you will run this on the cluster master. Note I have set “replication_count>3” which means it will list only buckets that have a number of copies above 3.

| rest /services/cluster/master/buckets filter=replication_count>3| rex field=title "^(?<repl_index>[^\~]+)"
   | search repl_index="*" standalone=0 frozen=0
   | rename title AS bucketID
   | fields bucketID peers.*.search_state  *site*
   | untable bucketID siteState value
   | rex field=siteState "peers\.(?<peerGUID>[^\.]*?)\.(?<siteState>search_state)"
   | rex field=siteState "(?<siteState>primaries_by_site)\.(?<site>\S+)"
   | rex field=siteState "(?<siteState>rep_count_by_site)\.(?<site>\S+)"
   | rex field=siteState "(?<siteState>search_count_by_site)\.(?<site>\S+)"
   | eval peerGUID=if(siteState=="primaries_by_site", value, peerGUID)
   | eval site=if(siteState=="origin_site", value, site)
   | eval value=if(siteState=="search_count_by_site", site + ":" + value, value)
   | eval value=if(siteState=="rep_count_by_site", site + ":" + value, value)
   | join type=outer peerGUID [ rest /services/cluster/master/peers
                          | fields active_* host* label title status site
                          | eval PeerName= site + ":" + label + ":" + host_port_pair
                          | rename title AS peerGUID
                          | rename site AS peerSite
                          | table peerGUID PeerName peerSite ]
   | eval site=if(siteState=="search_state", peerSite, site)
   | eval value=if(siteState=="primaries_by_site", PeerName + ":For_" + site, value)
   | eval value=if(siteState=="search_state", PeerName + ":" + value, value)
   | fields - PeerName peerGUID peerSite    | chart values(value) over bucketID by siteState 

For the list of the Buckets listed - check the Bucket REST end point using URL

https://:/services/cluster/master/buckets/

like.....

https://:/services/cluster/master/buckets/_audit~1~ABE7B836-1BD4-4EBD-8F2F-740DAE1DB9F4

check these buckets' REST end point for attribute "constrain_to_origin_site" . If this attribute has a value of '1' - it means that bucket was created before enabling multisite. These buckets won't be removed for excess copies.

View solution in original post

rbal_splunk
Splunk Employee
Splunk Employee

Run the search below to get the list of extra buckets - you will run this on the cluster master. Note I have set “replication_count>3” which means it will list only buckets that have a number of copies above 3.

| rest /services/cluster/master/buckets filter=replication_count>3| rex field=title "^(?<repl_index>[^\~]+)"
   | search repl_index="*" standalone=0 frozen=0
   | rename title AS bucketID
   | fields bucketID peers.*.search_state  *site*
   | untable bucketID siteState value
   | rex field=siteState "peers\.(?<peerGUID>[^\.]*?)\.(?<siteState>search_state)"
   | rex field=siteState "(?<siteState>primaries_by_site)\.(?<site>\S+)"
   | rex field=siteState "(?<siteState>rep_count_by_site)\.(?<site>\S+)"
   | rex field=siteState "(?<siteState>search_count_by_site)\.(?<site>\S+)"
   | eval peerGUID=if(siteState=="primaries_by_site", value, peerGUID)
   | eval site=if(siteState=="origin_site", value, site)
   | eval value=if(siteState=="search_count_by_site", site + ":" + value, value)
   | eval value=if(siteState=="rep_count_by_site", site + ":" + value, value)
   | join type=outer peerGUID [ rest /services/cluster/master/peers
                          | fields active_* host* label title status site
                          | eval PeerName= site + ":" + label + ":" + host_port_pair
                          | rename title AS peerGUID
                          | rename site AS peerSite
                          | table peerGUID PeerName peerSite ]
   | eval site=if(siteState=="search_state", peerSite, site)
   | eval value=if(siteState=="primaries_by_site", PeerName + ":For_" + site, value)
   | eval value=if(siteState=="search_state", PeerName + ":" + value, value)
   | fields - PeerName peerGUID peerSite    | chart values(value) over bucketID by siteState 

For the list of the Buckets listed - check the Bucket REST end point using URL

https://:/services/cluster/master/buckets/

like.....

https://:/services/cluster/master/buckets/_audit~1~ABE7B836-1BD4-4EBD-8F2F-740DAE1DB9F4

check these buckets' REST end point for attribute "constrain_to_origin_site" . If this attribute has a value of '1' - it means that bucket was created before enabling multisite. These buckets won't be removed for excess copies.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...