I'm not sure how it came to be, but it looks like my _internal index's cold path has changed. This left a bunch of cold buckets in the old location which seems unreferenced by any index as seen by | dbinspect index=*
.
As noted here:
https://answers.splunk.com/answers/108941/deleting-a-bucket.html#answer-108942
hot/warm/cold buckets should not be removed while the server is running, but in this case, it seems it can be done. Can someone confirm?
It would be interesting to see what you get when running -
splunk list excess-buckets _internal
Based on Remove excess bucket copies from the indexer cluster
You probably should be just fine to remove them...
The command resulted in "In handler 'clustermasterpeerindexes': master is not enabled on this node".