topic Re: Data Retention Policy in Getting Data In

Data Retention Policy

sourabh_v — Fri, 07 Feb 2014 11:22:46 GMT

Hi All,

I am trying to setup the data retention policy in a way that my data in hot db stays for 1 day, 10 GB of data in warm and 30 days in cold. I have defined frozenTimePeriodInSecs = 7776000. However i am not able to identify the parameters to be set for hot db and warm db. please suggest.

regards,
Sourabh

Re: Data Retention Policy

kristian_kolb — Fri, 07 Feb 2014 12:16:04 GMT

It's hard to make this kind of combination without knowing how much data you are actually indexing on a daily basis. There are simply no configuration parameters that fully implement such a retention policy. Also, it's a bit hard to understand the underlying requirements for such a policy.

Normally you would have a retention time requirement, say one year or 3 months, and possibly some constraint on the size or cost for fast/slow storage, which would force you to play around with when to move from warm to cold.

Given the docs for indexes.conf, I would suggest;

[your_index]
maxDataSize = 500
maxHotSpanSecs = 86400
homePath.maxDataSizeMB = 11000
maxTotalDataSizeMB = large number here, possibly larger than the default 500000
frozenTimePeriodInSecs = your actual retention time for all data

Re: Data Retention Policy

sourabh_v — Mon, 10 Feb 2014 06:09:24 GMT

Thanks Kristian for your input. indexs.conf spec file says that the maxHotSpanSecs will induce the snapping behavior and its for both hot/warm buckets. My requirement is
1. Data to be retained in hot bucket - 1 day (86400 secs)
2. Data in warm bucket = 300 GB
3. Data in Cold bucket = 30 Days

I am not sure how the snapping behavior would affect my data?

Re: Data Retention Policy

kristian_kolb — Mon, 10 Feb 2014 07:36:56 GMT

The point I was making is that your requirements don't add up - how do you know that you need 30 days of cold data, if you don't know how much data you have in hot/warm? If you're indexing 10 MB/day, the hot+warm storage would last for almost 10 years - then what's the point of another 30 days of cold (300 MB)?

If you index 150GB/day, the hot+warm lasts 2 days, and the cold storage would be almost 5 TB. These two extremes will change the storage needs quite a lot.

Normally, you'll have a retention time requirement for data that is online (hot+warm+cold) and offline (frozen).