How do I manually identify excess buckets in a multisite cluster?

[ashnet16_2]

Hello,

When trying to remove all excess buckets via the Cluster Master in a multisite indexer clustered environment, we don't see all excess buckets being removed, only some. Is it possible that the cluster master is only removing excess buckets from one site and not the other? Also, is there a way to identify excess buckets? Do excess buckets have a particular prefix? If so, is it save to remove them manually?

[s2_splunk · Accepted solution]

Excess buckets are the result of corrective action taken by the cluster master upon peer node failure to ensure that your configured replication factor is being met in the cluster. Because the cluster master at some point decided that certain buckets need to be replicated to meet your RF/SF, these buckets don't have any naming conventions that 'mark' them as excess buckets, they look like any other bucket. It is the fact that you have more copies of a given bucket than needed to satisfy RF/SF makes them 'excessive'. I strongly advise you to not try and take any manual action without involvement of Splunk support.
If you believe that the UI driven action does not remove all excess buckets AND your cluster is otherwise healthy, i.e. RF/SF are met and all peer nodes are up, please file a case with Splunk support.