Getting Data In

Why is old data not being deleted every day after configuring a 3 day retention period for an index?

Abilan1
Path Finder

Hi,

I am testing the retention related settings in my test index. I have set up the frozenTimePeriodInSecs = 259200. 259200 is for 3 days and it cleaned up all the 3 days old data after restart, but after that, everyday it is not cleaning up the data older than 3 days. Please let me know what could be the reason for this.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi @Abilan1

When forwarder sends data to Indexer it writes data in hot buckets, as per indexes.conf hot buckets rolls to Warm buckets when either maxHotSpanSecs(Default value is 90 days) or maxDataSize(Default value is auto means 750MB) reaches.

So if your hot buckets are not hitting any of the above parameter till that it will not roll to Warm buckets and splunk will not remove hot buckets because splunk removes only Warm and Cold buckets. If your hot buckets are not rolling to Warm/Cold buckets and it contains data then those data will be searchable.

And when you restarted splunk all your hot buckets rolls to Warm bucket and when all events in those warm buckets reaches retention period which you have given as 3 days was removed/deleted from splunk.

I hope this clears your query.

Thanks,
Harshil

Abilan1
Path Finder

Thanks for your update. Yes I can see only one hot bucket..Other than that I don't see any other buckets. But if I check in indexes, the earliest event date is 8 days back, that's not getting update. normally it get update after the next restart?

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

When you will restart splunk hot bucket will role to Warm bucket and splunk checks immediately that whether all events in that Warm bucket reached it's retention period if yes then splunk will remove that Warm bucket immediately. After that you will not able to see 8 days old data.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...