How to limit the max size of a single index?

[zpasplunk]

I have a bunch of indexes, but one in particular I want to keep smaller. How do I do this?

From the docs it looks like this could work?  Adding maxTotalDataSizeMB to the index config?

Anyknow know for sure?

/opt/splunk/etc/master-apps/_cluster/local/indexes.conf:

[webapp1]

maxTotalDataSizeMB = <nonnegative integer> * The maximum size of an index (in MB). * If an index grows larger than the maximum size, the oldest data is frozen. * This parameter only applies to hot, warm, and cold buckets. It does not apply to thawed buckets. * CAUTION: This setting takes precedence over other settings like 'frozenTimePeriodInSecs' with regard.....

[richgalloway · Accepted solution]

Yes, that is the knob to turn.

[zpasplunk]

After setting this, and pushing the cluster bundle to all nodes (they restart), what else do I have to do to get old data purged? I have a 500GB index which I now have set maxTotalDataSizeMB for that index to 10GB. Does it happen automatically, or do I have to do something else to shrink it back down?

[richgalloway]

When buckets are added to that index, Splunk will verify the max size has not been exceeded.  If it has, older buckets will be frozen until the index size is within the limit.

[zpasplunk]

Ok so the buckets have gone up by 1 and the data has gone up by 2GB. It's already way over my new limit. It doesn't appear to be honoring that, is there some other way to force this?

[jamie00171]

Hi @zpasplunk, how many indexers do you have in the cluster? I believe that maxTotalDataSizeMB is applied per indexer. 

Another way to remove old data from an index is to set:

frozenTimePeriodInSecs = <number of seconds before data is rolled to frozen>

One thing to note here is that each bucket contains data across a given time span (e.g. it could be 1 hour or 1 day) up to a maximum of the value set for

maxHotSpanSecs 

  The earliest timestamp of an event (oldest event) and latest timestamp of an event (most recent) are in the name of the bucket (https://docs.splunk.com/Documentation/Splunk/9.0.3/Indexer/HowSplunkstoresindexes#Bucket_naming_conv...).

Before a bucket is rolled to frozen the timestamp of the most recent event must be older than the value set for frozenTimePeriodInSecs which means there can be cases where there is data that's older than you'd expect. 

[zpasplunk]

Yes I was suspecting frozenTimePeriodInSecs as well based on the docs.. Thx