How do I configure and enforce a 6 month data retention policy?

[andrewtrobec]

Hello,

I am trying to configure a 6 month data retention policy in which data has to be deleted from an index 180 days after it has been indexed. Since buckets are defined based on the _time attribute of each event, _time is associated to the index time.

Now I know that buckets only get rolled once the newest event reaches the threshold, so it is important for me to configure one bucket per day so that I will always be sure to delete all data indexed on a given day on the 180th day

What I have so far in my indexes.conf for my index named retention is the following:

[retention]
coldPath = $SPLUNK_DB/retention/colddb
homePath = $SPLUNK_DB/retention/db
maxDataSize = 150 (this is set to the expected data consumption per day)
maxHotSpanSecs = 86400 (set to 1 day so that a bucket will be created per day)
maxTotalDataSizeMB = 27000 (set to 150 expected per day x 30 days per month x 6 months)
frozenTimePeriodInSecs = 15552000‬ (6 month retention before being frozen)

Would this be the correct configuration for my index? Is there some other parameter that I am missing?

Thanks!

Andrew

[viewsmart]

Your configuration looks correct. However, you've given no room for error margins, specifically with the frozenTimePeriodInSecs.
If you are using NTP within your environment, the protocol might experience errors, resulting in unexpecting behavior such as the deletion of your index data.

I'll advice you to increase the frozenTimePeriodInSecs above 6 months and allow the maxTotalDataSizeMB rule to enforce your retention policy.

[snowmizer]

frozenTimePeriodInSecs will force your retention settings on your index. In this case you are correct that your settings will force the data to roll off after 6 months.