topic Splunk Index archiving not effective in Splunk Enterprise

Splunk Index archiving not effective

jiaminyun — Mon, 20 Jan 2025 02:45:50 GMT

Hello, may I ask two questions
1) We are currently experiencing a 200 day archive configuration for the index, but it has not taken effect. Could you please advise on the triggering conditions for the frozenTimePeriodInsecs parameter.
2) Which is higher in priority between the frozenTimePeriodInsecs parameter of the index and maxTotalDataSizeMB?

Re: Splunk Index archiving not effective

kiran_panchavat — Mon, 20 Jan 2025 05:15:25 GMT

@jiaminyun

The priority between frozenTimePeriodInSecs and maxTotalDataSizeMB can be understood as follows:

maxTotalDataSizeMB Takes Precedence: If the index size exceeds

maxTotalDataSizeMB before reaching the time set in frozenTimePeriodInSecs, the data will be rolled to frozen state based on the size limit.

http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Setaretirementandarchivingpolicy

Re: Splunk Index archiving not effective

kiran_panchavat — Mon, 20 Jan 2025 05:17:32 GMT

@jiaminyun

Splunk prioritizes evaluating the total data size in the index against the `maxTotalDataSizeMB` parameter. If the total size exceeds the defined limit, Splunk will begin deleting the oldest buckets, regardless of whether they satisfy the retention period defined by `frozenTimePeriodInSecs`. Conversely, if the data size remains within the specified limit, the system will then assess buckets based on the `frozenTimePeriodInSecs` parameter to archive or delete those exceeding the time threshold. To ensure consistent data retention for a specific duration (e.g., 200 days), it is essential to configure `maxTotalDataSizeMB` to accommodate the anticipated volume of data for the desired retention period.

Re: Splunk Index archiving not effective

kiran_panchavat — Mon, 20 Jan 2025 05:36:47 GMT

@jiaminyun

If you find this solution satisfactory, please proceed to accept it.

Re: Splunk Index archiving not effective

jiaminyun — Mon, 20 Jan 2025 05:40:49 GMT

Your help was very much appreciated.

Re: Splunk Index archiving not effective

isoutamo — Mon, 20 Jan 2025 07:49:49 GMT

here is couple of links to old answers where we are discussed this.

Some of those are little bit out of direct scope of your question, but still those give to you better understanding how this is working.

r. Ismo

Re: Splunk Index archiving not effective

isoutamo — Mon, 20 Jan 2025 08:28:51 GMT

You must also remember that all time based activities has calculated on newest event in bucket. This is usually the reason why you have lot of of old events which should be archived by time. More about this on those links which I add on another post.

Re: Splunk Index archiving not effective

isoutamo — Mon, 20 Jan 2025 08:31:48 GMT

Not exactly that way. You must remember that all time based calculations has done by newest event on bucket! And you could have events e.g. within several months or even longer period (e.g. there is some reindexing for old data) in one bucket. See more from those links which I posted.

Re: Splunk Index archiving not effective

jiaminyun — Mon, 20 Jan 2025 08:44:28 GMT

谢谢。目前，假设我设置总索引大小为 500GB，实际使用了 140GB，配置的存档周期为 200 天，Hot/Arm/Guild Bucket 的最大大小设置为 auto-highvolume GB，但数据已经保留 4 年，仍然没有存档