Getting Data In

How to configure Splunk that data from all buckets (incl. frozen) older than a certain time are automatically deleted?

ddlliinn
New Member

According to documentation:

The maxTotalDataSizeMB and frozenTimePeriodInSecs attributes in indexes.conf help determine when buckets roll from cold to frozen, allowing you to configure data retention policy.
Data retention policy is applied only on Cold buckets. If maxTotalDataSizeMB is reached before frozenTimePeriodInSecs, data will be rolled to frozen before the configured time period has elapsed.
maxDataSize defines maximum size in MB for a hot DB to reach before a roll to warm is triggered. You should use "auto_high_volume" for high-volume indexes (such as the main index); otherwise, use "auto". A "high volume index" would typically be considered one that gets over 10GB of data per day.

In our environment, we have the following configuration for main index.

[main]
homePath = $SPLUNK_DB/defaultdb/db
coldPath = $SPLUNK_DB/defaultdb/colddb
thawedPath = $SPLUNK_DB/defaultdb/thaweddb
tstatsHomePath = volume:_splunk_summaries/defaultdb/datamodel_summary
maxMemMB = 20
maxConcurrentOptimizes = 6
maxHotIdleSecs = 86400
maxHotBuckets = 10
maxDataSize = auto_high_volume

and the following default values

frozenTimePeriodInSecs = 94348800 (3 years)
maxTotalDataSizeMB = 500000 (500G - default)

My main index size is 341.26 GB, so i would expect the frozenTimerPeriodInSecs to be applied.

However, the earliest event described on main index details page is aged back in 2014-03-13 18:58:01+0000.
Since it is the main index we have also the maxDataSize set to auto_high_volume, although the index gets aprox 1G data/day.
Could you please advise what could be wrong or misconfigured and the retention policy cannot be applied and data is not deleted?
Thank you in advance,

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @ddlliinn,
if you configure a value for the retention of all your indexes and you haven't a script to execute after Cold state, all the buckets with all events older than the retention value will be deleted.
If in a buchet you have all the events older than retention except one, bucket will not be deleted until the latest event exceeds the retention period.
If you have events older than retention period, surely they are in a bucket with events after the retention period.

Ciao.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...