Splunk Index archiving not effective

[jiaminyun]

Hello, may I ask two questions
1) We are currently experiencing a 200 day archive configuration for the index, but it has not taken effect. Could you please advise on the triggering conditions for the frozenTimePeriodInsecs parameter.
2) Which is higher in priority between the frozenTimePeriodInsecs parameter of the index and maxTotalDataSizeMB?

[isoutamo]

Hi

here is couple of links to old answers where we are discussed this.

• https://community.splunk.com/t5/Deployment-Architecture/Right-number-and-size-of-hot-warm-cold-bucke...
• https://community.splunk.com/t5/Deployment-Architecture/Hot-Warm-Cold-bucket-sizing-How-do-I-set-up-...
• https://community.splunk.com/t5/Deployment-Architecture/Index-rolling-off-data-before-retention-age/...
• https://community.splunk.com/t5/Splunk-Enterprise/Why-do-we-have-warm-buckets/m-p/700835

Some of those are little bit out of direct scope of your question, but still those give to you better understanding how this is working.

r. Ismo 

[kiran_panchavat]

@jiaminyun

If you find this solution satisfactory, please proceed to accept it.

[kiran_panchavat]

@jiaminyun 

Splunk prioritizes evaluating the total data size in the index against the `maxTotalDataSizeMB` parameter. If the total size exceeds the defined limit, Splunk will begin deleting the oldest buckets, regardless of whether they satisfy the retention period defined by `frozenTimePeriodInSecs`. Conversely, if the data size remains within the specified limit, the system will then assess buckets based on the `frozenTimePeriodInSecs` parameter to archive or delete those exceeding the time threshold. To ensure consistent data retention for a specific duration (e.g., 200 days), it is essential to configure `maxTotalDataSizeMB` to accommodate the anticipated volume of data for the desired retention period.

[isoutamo]

Not exactly that way. You must remember that all time based calculations has done by newest event on bucket! And you could have events e.g. within several months or even longer period (e.g. there is some reindexing for old data) in one bucket. See more from those links which I posted.

[jiaminyun]

谢谢。目前，假设我设置总索引大小为 500GB，实际使用了 140GB，配置的存档周期为 200 天，Hot/Arm/Guild Bucket 的最大大小设置为 auto-highvolume GB，但数据已经保留 4 年，仍然没有存档

[kiran_panchavat]

@jiaminyun  

The priority between frozenTimePeriodInSecs and maxTotalDataSizeMB can be understood as follows:

maxTotalDataSizeMB Takes Precedence: If the index size exceeds

maxTotalDataSizeMB before reaching the time set in frozenTimePeriodInSecs, the data will be rolled to frozen state based on the size limit.

http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Setaretirementandarchivingpolicy

[isoutamo]

You must also remember that all time based activities has calculated on newest event in bucket. This is usually the reason why you have lot of of old events which should be archived by time. More about this on those links which I add on another post.

[jiaminyun]

Your help was very much appreciated.