topic Re: How to limit the max size of a single index? in Getting Data In

How to limit the max size of a single index?

zpasplunk — Tue, 10 Jan 2023 17:39:22 GMT

I have a bunch of indexes, but one in particular I want to keep smaller. How do I do this?

From the docs it looks like this could work? Adding maxTotalDataSizeMB to the index config?

Anyknow know for sure?

/opt/splunk/etc/master-apps/_cluster/local/indexes.conf:

[webapp1]

maxTotalDataSizeMB = <nonnegative integer> * The maximum size of an index (in MB). * If an index grows larger than the maximum size, the oldest data is frozen. * This parameter only applies to hot, warm, and cold buckets. It does not apply to thawed buckets. * CAUTION: This setting takes precedence over other settings like 'frozenTimePeriodInSecs' with regard.....

Re: How to limit the max size of a single index?

richgalloway — Wed, 11 Jan 2023 01:32:17 GMT

Yes, that is the knob to turn.

Re: How to limit the max size of a single index?

zpasplunk — Fri, 20 Jan 2023 15:37:42 GMT

After setting this, and pushing the cluster bundle to all nodes (they restart), what else do I have to do to get old data purged? I have a 500GB index which I now have set maxTotalDataSizeMB for that index to 10GB. Does it happen automatically, or do I have to do something else to shrink it back down?

Re: How to limit the max size of a single index?

richgalloway — Fri, 20 Jan 2023 18:05:52 GMT

When buckets are added to that index, Splunk will verify the max size has not been exceeded. If it has, older buckets will be frozen until the index size is within the limit.

Re: How to limit the max size of a single index?

zpasplunk — Mon, 23 Jan 2023 17:46:50 GMT

Ok so the buckets have gone up by 1 and the data has gone up by 2GB. It's already way over my new limit. It doesn't appear to be honoring that, is there some other way to force this?

Re: How to limit the max size of a single index?

jamie00171 — Mon, 23 Jan 2023 18:37:08 GMT

Hi @zpasplunk, how many indexers do you have in the cluster? I believe that maxTotalDataSizeMB is applied per indexer.

Another way to remove old data from an index is to set:

frozenTimePeriodInSecs = <number of seconds before data is rolled to frozen>

One thing to note here is that each bucket contains data across a given time span (e.g. it could be 1 hour or 1 day) up to a maximum of the value set for

maxHotSpanSecs

The earliest timestamp of an event (oldest event) and latest timestamp of an event (most recent) are in the name of the bucket (https://docs.splunk.com/Documentation/Splunk/9.0.3/Indexer/HowSplunkstoresindexes#Bucket_naming_conventions).

Before a bucket is rolled to frozen the timestamp of the most recent event must be older than the value set for frozenTimePeriodInSecs which means there can be cases where there is data that's older than you'd expect.

Re: How to limit the max size of a single index?

zpasplunk — Mon, 23 Jan 2023 18:42:24 GMT

Yes I was suspecting frozenTimePeriodInSecs as well based on the docs.. Thx