topic Re: Settings for 1year retention period in Deployment Architecture

Settings for 1year retention period

jkamdar — Tue, 29 Jul 2025 19:54:51 GMT

I have this small Splunk Enterprise deployment in a lab that's air gapped.

So I setup this deployment about 18 months ago. Recently I noticed, I am not rolling any data. I want to set retention period of 1 year for all the data. After checking the configuration, looks like I have # of Hot buckets set to auto (which is 3 by default, I assume) but I don't find any Warm buckets. So, everything is in Hot buckets. I am looking at few settings maxHotSpanSecs, frozenTimePeriodInSecs and maxVolumeDataSizeMB, that should roll data to warm and then cold buckets eventually.

Under /opt/splunk/etc/system/local/indexes.conf
maxHotSpanSecs is set to 7776000
frozenTimePeriodInSecs 31536000
maxVolumeDataSizeMB (not set)

Under /opt/splunk/etc/apps/search/indexs.conf
maxHotSpanSecs not set
frozenTimePeriodInSecs 31536000 (for all the indexes)
maxVolumeDataSizeMB (not set)

Shouldn't frozenTimePeriodInSecs take precedent?

Maybe, my maxVolumeDataSizeMB is set to too high. Do I need to change it? How do frozenTimePeriodInSecs and maxVolumeDataSizeMB affect each other? I thought frozenTimePeriodInSecs would override maxVolumeDataSizeMB

Re: Settings for 1year retention period

livehybrid — Tue, 29 Jul 2025 21:02:58 GMT

Hi @jkamdar

Due to having things defined in different places here, it might be best to run a btool to the exact configuration (including default values)

Please could you run:

$SPLUNK_HOME/bin/splunk cmd btool indexes list --debug <yourIndexName>

When you talk about buckets "not rolling" - do you mean from Hot->Warm, or Cold->Frozen?

🌟 Did this answer help you? If so, please consider:

Adding karma to show it was useful
Marking it as the solution if it resolved your issue
Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

Re: Settings for 1year retention period

richgalloway — Tue, 29 Jul 2025 21:05:04 GMT

You say you found no warm buckets, but what about cold buckets? Did you find any of those?

If you're not running out of disk space then maxVolumeDataSizeMB is not too high.

The current settings have buckets spanning 90 days. Therefore, you should have 5 "generations" of buckets - 0-90 days (hot), 91-180 days, 181-270 days, 271-360 days, and 361-450 days. That last one is because a bucket won't be frozen until *all* events in it exceed the retention time.

Set maxHotSpanSecs to 86400 so each bucket only contains a single day of data and retention should improve.

Re: Settings for 1year retention period

jkamdar — Wed, 30 Jul 2025 17:02:55 GMT

Thanks @livehybrid and @richgalloway both suggestions are helpful. I was able to use btool to find what indexes.conf each index is using and then I did change maxHotSpanSecs to the suggested # and I see more warm buckets. If this going to trigger data deletion that's over an year old, that's great, I will wait and see.

However, regardless what was set for maxHotSpanSecs, shouldn't frozenTimePeriodInSecs have triggered the expiration of data and delete? O

I sure am not clear how maxHotSpanSecs and frozenTimePeriodInSecs work together and affects the retention period. If one can explain, it would be great.

Re: Settings for 1year retention period

richgalloway — Wed, 30 Jul 2025 18:10:18 GMT

The frozenTimePeriodInSecs setting does not apply to hot buckets. You should, however, see warm buckets once a hot bucket fills up or becomes 90 days old. I can't explain why you don't.

Re: Settings for 1year retention period

jkamdar — Wed, 06 Aug 2025 18:17:17 GMT

Ok, I have set maxHotSpanSecs 86400. I am seeing lots of warm buckets now. With and frozenTimePeriodInSecs 31536000, I think, I am seeing results that I was hoping for.

Also, with the below search, I am seeing data being rolled to frozen as well.

index=_internal sourcetype=splunkd log_level=INFO component=BucketMover "freeze succeeded"