What happens if maxTotalDataSizeMB is not declared?

[jessieb_83]

I understand that maxTotalDataSizeMB takes precedence over frozenTimePeriodInSecs.

What happens if frozenTimePeriodInSecs is defined and maxTotalDataSizeMB  is not? The Splunk docs don't cover this specific circumstance, and I haven't been able to find anything else about it.

I have requirement to keep all department security logs for 5 years regardless how big the indexes get. They need to delete at 5.1 years. My predecessor set it up so that frozenTimePeriodInSecs= (5.1years in seconds) and maxTotalDataSizeMB =1000000000mb (roughly 1000TB's) so that size would not affect retention, but now nothing will delete and we're retaining logs from 8 years ago.

If I comment out  maxTotalDataSizeMB, will frozenTimePeriodInSecs take precedence or will the default  maxTotalDataSizeMB settings take over? My indexes are roughly 7Tb, so the 500Gb would wipe out a bunch of stuff I need to keep. 

In my lab environment, I commented out  maxTotalDataSizeMB and set frozenTimePeriodInSecs to 6 months, but still have logs from 2 years ago. Unfortunately, my lab environment doesn't have enough archived data to test the default cut off. 

Thanks!

[jessieb_83]

Had to step away from this for a while due to more pressing fires. Finally got to look at it today, borrowed a nifty query from dbinspect query help - Splunk Community and found out that Splunk thinks my oldest bucket is from May 5, 2024, despite some being up to 8 years old. If I search against time, they seem to come up correctly going back years so, I'm at an utter loss on that one. 

But, at least I know why it's not rolling anything off now!

[JohnEGones]

Can you provide feedback on Rich's suggestion:

Use the dbinspect command to examine your buckets.  Make sure the oldest ones don't have an earliest_time that is newer than the frozenTimePeriodInSecs setting.  Buckets will not age out until *all* of the events in the bucket are old enough.

[richgalloway]

Splunk does not delete individual events - it removes entire buckets when either the size or time limit is reached.

When deleting by time, because the whole bucket is deleted, it's important that all of the events in that bucket be old enough to delete.  If any event is too new then the bucket will not be touched.  Every bucket has two dates (for our purposes, anyway) associated with it - the start date (_time of the first event added) and the end date (_time of the last event added).  The end date is one that determines when the bucket can be deleted/frozen.

I've seen sites where data is poorly onboarded and has _time values in the future - sometimes by years.  When that happens, the bucket will remain in the system until frozenTimePeriodInSecs after that future date passes.

[richgalloway]

---

@jessieb_83 wrote:

I understand that maxTotalDataSizeMB takes precedence over frozenTimePeriodInSecs.

You understand incorrectly.  Both have equal precedence.  The first limit reached is the one that will be applied.  If the size limit is reached then the oldest buckets will be deleted first.

If maxTotalDataSizeMB is not specified, it defaults to 500000 (see indexes.conf.spec).

Use the dbinspect command to examine your buckets.  Make sure the oldest ones don't have an earliest_time latest_time (endEpoch) that is newer than the frozenTimePeriodInSecs setting.  Buckets will not age out until *all* of the events in the bucket are old enough.

[jessieb_83]

Rich, thanks for the clarification. The Splunk documentation is kinda confusing on this specific topic. 

That is helpful, frustraiting and leaves me with even more questions. Now I have absolutely no idea why we have logs 3 years older than the retentions are set to. There is nothing set up to freeze anything so it should all be rolling out as it hits that 5.1 year mark.

Would homePath.maxDataSizeMB override? I thought that was the cut off to roll warm to cold and shouldn't affect it. 

The only limits set are:

maxTotalDataSizeMB = 1000000000 #1,000TB

homePath.maxDataSizeMB = 500000 #500GB

frozenTimePeriodInSecs = 160833600 (5.1 years)

maxDataSize = 2000

maxWarmDBCount = 2000