Multisite Indexer Cluster Inconsistent Buckets: How to delete buckets and their metadata to no longer be listed in the cluster master REST endpoint?

[tluluma3]

Every time when I do a splunk rolling-restart cluster-Peers I have some buckets that are listed in the REST Endpoint, but they are not on the filesystem on the particular Indexer.
I can find these buckets with:

index=_internal host="CLUSTERMASTER" sourcetype=splunkd from=Unsearchable | stats values(peer_name) by bid

I then can delete the buckets with splunk _internal call /services/cluster/master/buckets/BID/remove_all -method POST, but they reappear after the next rolling restart.

Is there a way to delete the metadata from these buckets for the Cluster master to "forget" them forever?

[dxu_splunk]

seems like you want to delete a bucket entirely.

if remove_all doesn't work (it might not work if the bucket is in some transitory state, ie PendingSearchable), you can try freezing the bucket on the indexers.

curl -k -u USER:PASS https://indexer:mgmt_port/services/data/indexes/INDEX/freeze-buckets -d bucket_ids=117_22220097-5E3F-4D26-B301-ECE3C4CD2222 -X POST

bucket_ids is a list of buckets by ID_GUID. this should remove the bucket on that specific indexer - just repeat for all the other copies of the bucket

[dxu_splunk]

can you post the logs for a specific bucket ?

[tluluma3]

10-15-2015 13:48:43.465 +0200 INFO CMPeer - peer=7EBE2435-3FB3-4CE2-85E3-35931CD26293 peer_name=zhhapssp-slin05.it.bwns.ch bid=_internal~11942~14EE1C69-1AED-4A51-8A8E-0FDCBE2C16FC transitioning from=Unsearchable to=PendingSearchable oldmask=0x0 newmask=0x0 reason="fixup searchable count"
10-15-2015 13:42:08.936 +0200 INFO CMPeer - peer=DF9EF4CB-14CE-4F5B-8BA0-9D687671B4B6 peer_name=oltapssp-slin02.it.bwns.ch bid=_internal~11942~14EE1C69-1AED-4A51-8A8E-0FDCBE2C16FC transitioning from=Unsearchable to=PendingSearchable oldmask=0x0 newmask=0x0 reason="fixup searchable count"
10-15-2015 13:46:07.692 +0200 INFO CMPeer - peer=DF9EF4CB-14CE-4F5B-8BA0-9D687671B4B6 peer_name=oltapssp-slin02.it.bwns.ch bid=_internal~11942~14EE1C69-1AED-4A51-8A8E-0FDCBE2C16FC transitioning from=Unsearchable to=PendingSearchable oldmask=0x0 newmask=0x0 reason="fixup searchable count"
10-15-2015 13:39:28.442 +0200 INFO CMPeer - peer=7EBE2435-3FB3-4CE2-85E3-35931CD26293 peer_name=zhhapssp-slin05.it.bwns.ch bid=_internal~11942~14EE1C69-1AED-4A51-8A8E-0FDCBE2C16FC transitioning from=Unsearchable to=PendingSearchable oldmask=0x0 newmask=0x0 reason="fixup searchable count"
10-15-2015 13:36:19.039 +0200 INFO CMPeer - peer=DF9EF4CB-14CE-4F5B-8BA0-9D687671B4B6 peer_name=oltapssp-slin02.it.bwns.ch bid=_internal~11942~14EE1C69-1AED-4A51-8A8E-0FDCBE2C16FC transitioning from=Unsearchable to=PendingSearchable oldmask=0x0 newmask=0x0 reason="fixup searchable count"
10-15-2015 13:36:00.952 +0200 INFO CMPeer - peer=7EBE2435-3FB3-4CE2-85E3-35931CD26293 peer_name=zhhapssp-slin05.it.bwns.ch bid=_internal~11942~14EE1C69-1AED-4A51-8A8E-0FDCBE2C16FC transitioning from=Unsearchable to=PendingSearchable oldmask=0x0 newmask=0x0 reason="fixup searchable count"