Getting Data In

Data Retention Policy

sourabh_v
Explorer

Hi All,

I am trying to setup the data retention policy in a way that my data in hot db stays for 1 day, 10 GB of data in warm and 30 days in cold. I have defined frozenTimePeriodInSecs = 7776000. However i am not able to identify the parameters to be set for hot db and warm db. please suggest.

regards,
Sourabh

Tags (2)

kristian_kolb
Ultra Champion

It's hard to make this kind of combination without knowing how much data you are actually indexing on a daily basis. There are simply no configuration parameters that fully implement such a retention policy. Also, it's a bit hard to understand the underlying requirements for such a policy.

Normally you would have a retention time requirement, say one year or 3 months, and possibly some constraint on the size or cost for fast/slow storage, which would force you to play around with when to move from warm to cold.

Given the docs for indexes.conf, I would suggest;

[your_index]
maxDataSize = 500
maxHotSpanSecs = 86400
homePath.maxDataSizeMB = 11000
maxTotalDataSizeMB = large number here, possibly larger than the default 500000
frozenTimePeriodInSecs = your actual retention time for all data

/k

kristian_kolb
Ultra Champion

The point I was making is that your requirements don't add up - how do you know that you need 30 days of cold data, if you don't know how much data you have in hot/warm? If you're indexing 10 MB/day, the hot+warm storage would last for almost 10 years - then what's the point of another 30 days of cold (300 MB)?

If you index 150GB/day, the hot+warm lasts 2 days, and the cold storage would be almost 5 TB. These two extremes will change the storage needs quite a lot.

Normally, you'll have a retention time requirement for data that is online (hot+warm+cold) and offline (frozen).

0 Karma

sourabh_v
Explorer

Thanks Kristian for your input. indexs.conf spec file says that the maxHotSpanSecs will induce the snapping behavior and its for both hot/warm buckets. My requirement is
1. Data to be retained in hot bucket - 1 day (86400 secs)
2. Data in warm bucket = 300 GB
3. Data in Cold bucket = 30 Days

I am not sure how the snapping behavior would affect my data?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...