For data retention what takes precedence, time or disk usage?

[scottj1y]

We have an index with a retention set to 6 hours or 300 GB of disk space. The index is only 46 GB right now but there are events that are days old still in it. Why haven't they been aged out?

The configuration change was pushed out by the master indexer which did a rolling restart of the peer indexers. Here's the configuration file

[main]
repFactor=auto
homePath = $SPLUNK_DB/main/db
coldPath = $SPLUNK_DB/main/colddb
thawedPath = $SPLUNK_DB/main/thawddb
# Set maximum data size of index to 300,000MB/300GB
maxTotalDataSizeMB = 300000
# Set maximun retention to 6 hrs
frozenTimePeriodInSecs = 21600


[long_lived]
repFactor=auto
homePath = $SPLUNK_DB/long_lived/db
coldPath = $SPLUNK_DB/long_lived/colddb
thawedPath = $SPLUNK_DB/long_lived/thaweddb
# Set maximum data size of index to 700,000MB/700GB
maxTotalDataSizeMB = 700000

[niketn]

Max size overrides all other retention settings.

[woodcock]

Show your settings to get better help. Did you restart the splunk instances on your indexers after you deployed your changes?

[woodcock]

All settings for retention are applied independently; there is no precedence/cooperation.

[jhupka_splunk]

It is also important to consider that if you are often hitting the maxTotalDataSizeMB before the age specified in frozenTimePeriodInSecs, then you can potentially confusing "holes" in your data when people search. This especially becomes apparent with multiple Indexers because Indexer 1 might start running out of disk and culling buckets, but Indexer 2 is fine on space and has older data spanning the time period of the deleted buckets from Indexer 1. When you search over the time period, there might be end-user confusion if they are expecting 100% of the results for that time period but due to size half were deleted on Indexer 1.

[woodcock]

Yes, it is VERY important that each indexer gets the same amount of data for each index or holes will result.

[scottj1y]

There seems to be some disagreement. Woodcock you say there is no procedence and Niketnilay says max size overrides all other retention settings.

[woodcock]

I agree with everybody; it is just different ways of saying the same thing. There is NO setting that can cause another setting that has been triggered NOT to act.

[jkat54]

If guns don't kill people, people kill people, does that mean toast toast toast? 😛

I feel like that's what we are saying.

[jkat54]

I agree with woodcock. Whichever comes first takes prescedence be it time or size... Other way of saying it is that all take precedence.