topic Re: Retention Issue: Why is data getting rolled to frozen before hitting the frozenTimePeriodInSecs setting? in Getting Data In

Retention Issue: Why is data getting rolled to frozen before hitting the frozenTimePeriodInSecs setting?

arber — Mon, 28 Sep 2020 18:04:11 GMT

Hello,
we are currently having some issues with an index. Basically we have configured the following in the related index:
[juniper_nsm]
coldPath = $SPLUNK_DB/juniper_nsm/colddb
coldToFrozenDir = $SPLUNK_DB/juniper_nsm/frozendb
homePath = $SPLUNK_DB/juniper_nsm/db
maxDataSize = auto_high_volume
maxTotalDataSizeMB = 600000
thawedPath = $SPLUNK_DB/juniper_nsm/thaweddb
frozenTimePeriodInSecs = 15552000

Theoretically with that frozen time period we should have the logs for at least 6 months online and once frozen moved to the frozendb as per configuration. The problem is that the logs get frozen before that period, its even less that 3 months. we can find the buckets to the frozendb. The index is not full and there is no other configuration for this index in the system.
One thing to mention is that this system generates a huge amount of logs nearly 20-25 gb per day.
What can be the issue that the logs get frozen before the frozentimeperiodsinsec ??Anything related to the maximum buckets in the db ? maybe if a certain nr of buckets is reached it has a larger priority than frozentimeperiod so the logs get frozen ?
Any idea ??

Thanks

Re: Retention Issue: Why is data getting rolled to frozen before hitting the frozenTimePeriodInSecs setting?

acharlieh — Mon, 03 Nov 2014 15:31:31 GMT

Your maxTotalDataSizeMB setting is 600000 (roughly 585 GB) . If you're logging 20 GB / day to that index, that means your buckets will roll to frozen in roughly 29 days. Buckets can be rolled to frozen if either maxTotalDataSizeMB or frozenTimePeriodInSecs is met. Check out the indexes.conf docs.

Re: Retention Issue: Why is data getting rolled to frozen before hitting the frozenTimePeriodInSecs setting?

arber — Mon, 03 Nov 2014 16:22:47 GMT

Thanks for the answer.. i will increment it . But i haven't seen this index full or nearly full .. we have an alerting policy in place that if the size is more then 90 % we get an alert. anyway i will increment this and see what happens

Thanks again

Re: Retention Issue: Why is data getting rolled to frozen before hitting the frozenTimePeriodInSecs setting?

vaithi_m — Mon, 01 Feb 2016 14:19:04 GMT

Is it possible to leave 'maxTotalDataSizeMB' un configured or to set it unlimited, so that the 'frozenTimePeriodInSecs' alone can determine the data frozeezing behavior?

Re: Retention Issue: Why is data getting rolled to frozen before hitting the frozenTimePeriodInSecs setting?

baldwintm — Mon, 08 Feb 2016 14:41:31 GMT

if you set it to the Max Value (4294967295 = 4 Petabytes), then you should never hit that.

Re: Retention Issue: Why is data getting rolled to frozen before hitting the frozenTimePeriodInSecs setting?

vaithi_m — Thu, 18 Feb 2016 12:54:06 GMT

Hi baldwintm, Thanks for your suggestion. It really helped.