Makes sense. I thought the underlying storage engine for the indexes is MongoDB, which is why I was thinking there must be a way to reclaim space. We can wait for data to roll off, but in our case we have a long retention period (1-2yr +), necessary for performing investigations over the past couple of years. Combined with hardened boxes that generate a lot of log data to begin with. The two set up a situation where you can get a lot of undesired noise in the indexes before you realize it, and then you're stuck with a storage issue for a couple years while you wait for it to roll off because there is other good data in the index that you do want to retain. There must be a way to clean up what amounts to a "data spill" in your indexes while retaining the good data. The documentation for the 'delete' command indicates it only removes the data from regular searches, leaves references to it in metadata searches, and never reclaims the space. Is it marking the locations for overwrite? Will it be used by other data as long as the bucket is still active? Thanks.
... View more