Why is old data not being deleted every day after configuring a 3 day retention period for an index?

[Abilan1]

Hi,

I am testing the retention related settings in my test index. I have set up the frozenTimePeriodInSecs = 259200. 259200 is for 3 days and it cleaned up all the 3 days old data after restart, but after that, everyday it is not cleaning up the data older than 3 days. Please let me know what could be the reason for this.

[harsmarvania57]

Hi @Abilan1

When forwarder sends data to Indexer it writes data in hot buckets, as per indexes.conf hot buckets rolls to Warm buckets when either maxHotSpanSecs(Default value is 90 days) or maxDataSize(Default value is auto means 750MB) reaches.

So if your hot buckets are not hitting any of the above parameter till that it will not roll to Warm buckets and splunk will not remove hot buckets because splunk removes only Warm and Cold buckets. If your hot buckets are not rolling to Warm/Cold buckets and it contains data then those data will be searchable.

And when you restarted splunk all your hot buckets rolls to Warm bucket and when all events in those warm buckets reaches retention period which you have given as 3 days was removed/deleted from splunk.

I hope this clears your query.

Thanks,
Harshil

[Abilan1]

Thanks for your update. Yes I can see only one hot bucket..Other than that I don't see any other buckets. But if I check in indexes, the earliest event date is 8 days back, that's not getting update. normally it get update after the next restart?

[harsmarvania57]

When you will restart splunk hot bucket will role to Warm bucket and splunk checks immediately that whether all events in that Warm bucket reached it's retention period if yes then splunk will remove that Warm bucket immediately. After that you will not able to see 8 days old data.