Every time when I do a splunk rolling-restart cluster-Peers I have some buckets that are listed in the REST Endpoint, but they are not on the filesystem on the particular Indexer.
I can find these buckets with:
index=_internal host="CLUSTERMASTER" sourcetype=splunkd from=Unsearchable | stats values(peer_name) by bid
I then can delete the buckets with splunk _internal call /services/cluster/master/buckets/BID/remove_all -method POST, but they reappear after the next rolling restart.
Is there a way to delete the metadata from these buckets for the Cluster master to "forget" them forever?