Retention Issue: Why is data getting rolled to frozen before hitting the frozenTimePeriodInSecs setting?

[arber]

Hello,
we are currently having some issues with an index. Basically we have configured the following in the related index:
[juniper_nsm]
coldPath = $SPLUNK_DB/juniper_nsm/colddb
coldToFrozenDir = $SPLUNK_DB/juniper_nsm/frozendb
homePath = $SPLUNK_DB/juniper_nsm/db
maxDataSize = auto_high_volume
maxTotalDataSizeMB = 600000
thawedPath = $SPLUNK_DB/juniper_nsm/thaweddb
frozenTimePeriodInSecs = 15552000

Theoretically with that frozen time period we should have the logs for at least 6 months online and once frozen moved to the frozendb as per configuration. The problem is that the logs get frozen before that period, its even less that 3 months. we can find the buckets to the frozendb. The index is not full and there is no other configuration for this index in the system.
One thing to mention is that this system generates a huge amount of logs nearly 20-25 gb per day.
What can be the issue that the logs get frozen before the frozentimeperiodsinsec ??Anything related to the maximum buckets in the db ? maybe if a certain nr of buckets is reached it has a larger priority than frozentimeperiod so the logs get frozen ?
Any idea ??

Thanks

[acharlieh]

Your maxTotalDataSizeMB setting is 600000 (roughly 585 GB) . If you're logging 20 GB / day to that index, that means your buckets will roll to frozen in roughly 29 days. Buckets can be rolled to frozen if either maxTotalDataSizeMB or frozenTimePeriodInSecs is met. Check out the indexes.conf docs.

[arber]

Thanks for the answer.. i will increment it . But i haven't seen this index full or nearly full .. we have an alerting policy in place that if the size is more then 90 % we get an alert. anyway i will increment this and see what happens

Thanks again

[vaithi_m]

Is it possible to leave 'maxTotalDataSizeMB' un configured or to set it unlimited, so that the 'frozenTimePeriodInSecs' alone can determine the data frozeezing behavior?

[baldwintm]

if you set it to the Max Value (4294967295 = 4 Petabytes), then you should never hit that.

[vaithi_m]

Hi baldwintm, Thanks for your suggestion. It really helped.