What happens to a bucket if all of the data within it is deleted?

[caseyra]

Just what the title says. If I delete a bunch of data from an index and some of the buckets are now effectively empty (E.g. all of them have been marked as deleted), what happens to the bucket? Is it removed or will it stick around until it is frozen?

[s2_splunk · Accepted solution]

It will stick around until the bucket is frozen. The bucket mover uses the epoch times in the bucket (directory) name and doesn't really inspect the contents of the bucket to determine whether it can be frozen or not.

[caseyra]

O.K. So, is there a way to determine if a bucket has only deleted entries? And, if so, can we force the bucket to be rolled to frozen? Or, is the file basically stuck there until it is frozen?

[s2_splunk]

The short answer is no, buckets will honor the configured retention policy settings and there is no way to force a freeze unless you are (temporarily) reducing the index configuration. I am not aware of any way to safely determine whether a given bucket contains only deleted events.
The only way currently to physically delete events (buckets, really), is to run clean eventdata, which will wipe everything for the given index, so you'll have to come up with a plan to export/collect any events you do not want to delete.

What's your use case? Compliance? New EU regulations?

[caseyra]

Follow-up question: I did a little digging with dbinspect and noticed that some buckets can have an eventCount of zero, but the rawSize and/or sizeOnDiskMB is non-zero. This seems to indicate that these buckets have had all their records deleted, but are still taking up space on the system. Is that correct?

[caseyra]

It's more of a data cleanup issue. I was trying to avoid having to re-index a large chunk of data.

Thanks.

[ChrisG]

It depends what you did to delete the data: did you use the delete command (which does not actually remove anything from disk), or clean, or remove index?

If you have not already done so, I highly encourage you to read Remove indexes and indexed data in the Splunk Enterprise documentation.

[caseyra]

Just delete. I'm not trying to remove the entire index. I just wanted to know what would happen from a disk space perspective.