Knowledge Management

Clarification on indexer retention

mcbradford
Contributor

The documentation on this topic is not clear, so I am hoping someone can answer this for me. I need to keep data for 13 months, there are two settings within indexer.conf, frozenTimePeriodinSecs and maxTotalDataSizeMB. What is not clear is who wins? If I set maxTotalDataSizeMB to 1, but frozenTimePeriodinSeconds to 34187400, and my data is larger than 1mb, will my data roll over it reaches 1mb? Can I remove or comment out the stanza maxTotalDataSizeMB to force Splunk to just use the frozenTimePeriodinSeconds stanza?

Tags (1)
0 Karma
1 Solution

inventsekar
SplunkTrust
SplunkTrust

I need to keep data for 13 months, there are two settings within indexer.conf, frozenTimePeriodinSecs and maxTotalDataSizeMB.
What is not clear is who wins? If I set maxTotalDataSizeMB to 1, but frozenTimePeriodinSeconds to 34187400, and my data is larger than 1mb, will my data roll over it reaches 1mb? ///

Yes. Splunk will freeze data when either of maxtotaldatasizeMB or frozenTimePeriodInSecs is reached first.

View solution in original post

jkat54
SplunkTrust
SplunkTrust

Whichever comes first wins. However, frozenTimePeriodInSecs requires the entire bucket to be older than the frozenTimePeriodInSecs. Whereas maxTotalDataSizeMB will freeze the oldest bucket(s) once it's limit is hit. This is outlined in indexes.conf documentation:

maxTotalDataSizeMB = <nonnegative integer>
* The maximum size of an index (in MB).
* If an index grows larger than the maximum size, the oldest data is frozen.
* This parameter only applies to hot, warm, and cold buckets.  It does not
  apply to thawed buckets.
* Highest legal value is 4294967295
* Defaults to 500000.

frozenTimePeriodInSecs = <nonnegative integer>
* Number of seconds after which indexed data rolls to frozen.
* If you do not specify a coldToFrozenScript, data is deleted when rolled to
  frozen.
* IMPORTANT: Every event in the DB must be older than frozenTimePeriodInSecs
  before it will roll. Then, the DB will be frozen the next time splunkd
  checks (based on rotatePeriodInSecs attribute).
* Highest legal value is 4294967295
* Defaults to 188697600 (6 years).

You cant just remove the maxTotalDataSizeMB because it will always default to 500GB.

0 Karma

inventsekar
SplunkTrust
SplunkTrust

I need to keep data for 13 months, there are two settings within indexer.conf, frozenTimePeriodinSecs and maxTotalDataSizeMB.
What is not clear is who wins? If I set maxTotalDataSizeMB to 1, but frozenTimePeriodinSeconds to 34187400, and my data is larger than 1mb, will my data roll over it reaches 1mb? ///

Yes. Splunk will freeze data when either of maxtotaldatasizeMB or frozenTimePeriodInSecs is reached first.

Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...