We have a multisite indexer cluster running Splunk 6.2.7. with 40 indexers.
Many of the indexers have their data partitions over 90% full.
We are trying to clear up disk space in order to avoid catastrophic outage of the indexer cluster.
Ultimately, we are planning to add new indexers to the indexing pool, but we need an interim solution to buy us time.
Our primary site is site 2 while our DR site is site 1.
We want to reduce the replication factor by one.
The current replication settings on the cluster master is below:
After the config change to server.conf on the cluster master, what is the procedure to apply and remove the excess buckets?
* server.conf change on cluster master
* restart splunk on cluster master to apply change
* Click "Remove All Excess Buckets" button in Bucket Status view on the cluster master ui
* Wait for excess buckets to be deleted
Here are Bugs around "remove excess-buckets" that you might be interested in.
SPL-108023/ SPL-98101 :[Clustering] "remove excess-buckets" removes all buckets created under Multi-Site clustering if CM was moved from Multisite to Single Site. ( Reported on : 6.2.4, 6.2.6 and fixed on 6.2.7 and 6.3)
SPL-90409 :remove excess buckets does not remove all the excess buckets it should and causes "fully searchable" criteria in UI to fail ( This is not a majore buck: remove-excess-buckets may cause all-searchable to become not-all-searchable, but it should quickly return to all-searchable) - fixed in 6.3
SPL-106614 :remove excess bucket doesn't remove summary files (this bug is still pending) SPL-90986:splunk list excess-buckets only lists 30 indexes (Reported on 6.1.3 and Resolved in 6.3) SPL-98007:Clicking "remove" button in the cluster management excess buckets page deletes all standalone buckets (Reported on 6.2.1, 6.2.2 and fixed in 6.2.3)