Questions about data retention in Splunk Cloud (DDAS, DDAA)

[azer271]

Hello! I'm new to Splunk Cloud. Could you please explain the difference between hot, warm, cold and thawed buckets in Splunk Enterprise and Splunk Cloud? I understand that in Splunk Enterprise, a bucket moves through several states (from hot to thawed).

However, when I click on a new index in Splunk Cloud, I only saw "Searchable retention (days)" and "Dynamic Data Storage". Does this mean that the amount of data that can be searched in the hot and warm buckets before it goes to cold is basically equal to the searchable retention (days)? Does Dynamic Data Storage basically equate to the Cold, Frozen and Thawed buckets (as in Splunk Enterprise)?

Furthermore, in the Splunk Cloud Monitoring Console, I can see DDAS and DDAA in the 'License' section. What exactly are these, and what is their relationship with data retention? What happens if the DDAS/DDAA exceeds 100%? Does this affect searching performance, or does Splunk Cloud simply not allow you to search data? Thanks.

[livehybrid · Accepted solution]

Hi @azer271 

In Splunk Cloud, bucket management is abstracted and handled by Splunk therefore users do not directly interact with hot/warm/cold distinctions. Instead the storage usage is based on the raw ingested volumes.

DDAS (Dynamic Data Active Storage)  is the equiv. to your hot/warm/cold buckets from Splunk Enterprise, and can be configured as to how long data remains in this active, fast-searchable storage.

DDAA (Dynamic Data Active Archive)  is an additional license cost and is essentially a bit like frozen bucket storage - there is a mechanism in Splunk Cloud to restore this data (which isnt instant) which remains searchable for a period of time before being removed again (and retained in DDAA). This can be cost effective but also tricky to manage, if you need to search the data you will need to know what timeframe you need the data from when you restore it.

Another storage type is Dynamic Data: Self-Storage (DDSS) which is the equiv of a frozen bucket storage within your own S3 buckets. This isnt restorable back in to Splunk Cloud so if you ever needed to restore it then you'd need to do this back to your own on-premise Splunk Enterprise instance to thaw it out and become searchable again.

The Cloud Monitoring Console makes it easy to see your DDAS/DDAA usage, If DDAS/DDAA exceeds 100% then you may be liable for overage costs. It usually doesnt impact performance because of the elastic nature of the backend storage, but over-consuming and then searching more data than scoped can slow things down.

For more info its worth checking out https://www.splunk.com/en_us/blog/platform/dynamic-data-data-retention-options-in-splunk-cloud.html

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[azer271]

Thank you for your help!

[PrewinThomas]

@azer271 

I'm sure you are well aware how buckets moves in Splunk Enterprise. 

DDAS (Dynamic Data Active Searchable)
Equivalent to hot + warm + cold buckets. Data is searchable and stored in Splunk-managed infrastructure. Also controlled by the Searchable Retention (days) setting per index.

DDAA (Dynamic Data Active Archive)
Equivalent to frozen buckets. Data is archived but can be restored for search (up to 30 days) and it's managed by Splunk

Please note that there is a restoration limit - up to 10% of your DDAS entitlement at any time

So, the Searchable Retention defines how long data remains in the searchable tier (hot/warm/cold equivalent), and Dynamic Data Storage handles what happens after that.

Also if you exceed 100%, Splunk elastically expand DDAS to retain data, but consistent overages can impact search performance and potentially additional cost as well.

You can refer below for detailed info.
#https://splunk.my.site.com/customer/s/article/Details-for-DDAS-and-DDAA

#https://www.splunk.com/en_us/blog/platform/dynamic-data-data-retention-options-in-splunk-cloud.html?...

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[azer271]

Thank you for your help! The provided URLs are very useful.