Getting Data In

How to limit the max size of a single index?

zpasplunk
Explorer

I have a bunch of indexes, but one in particular I want to keep smaller. How do I do this?

From the docs it looks like this could work?  Adding maxTotalDataSizeMB to the index config?

Anyknow know for sure?

/opt/splunk/etc/master-apps/_cluster/local/indexes.conf:

[webapp1]
maxTotalDataSizeMB = <nonnegative integer> * The maximum size of an index (in MB). * If an index grows larger than the maximum size, the oldest data is frozen. * This parameter only applies to hot, warm, and cold buckets. It does not apply to thawed buckets. * CAUTION: This setting takes precedence over other settings like 'frozenTimePeriodInSecs' with regard.....
Labels (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Yes, that is the knob to turn.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

Yes, that is the knob to turn.

---
If this reply helps you, Karma would be appreciated.

zpasplunk
Explorer

After setting this, and pushing the cluster bundle to all nodes (they restart), what else do I have to do to get old data purged? I have a 500GB index which I now have set maxTotalDataSizeMB for that index to 10GB. Does it happen automatically, or do I have to do something else to shrink it back down?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

When buckets are added to that index, Splunk will verify the max size has not been exceeded.  If it has, older buckets will be frozen until the index size is within the limit.

---
If this reply helps you, Karma would be appreciated.
0 Karma

zpasplunk
Explorer

Ok so the buckets have gone up by 1 and the data has gone up by 2GB. It's already way over my new limit. It doesn't appear to be honoring that, is there some other way to force this?

Tags (1)
0 Karma

jamie00171
Communicator

Hi @zpasplunk, how many indexers do you have in the cluster? I believe that maxTotalDataSizeMB is applied per indexer. 

Another way to remove old data from an index is to set:

frozenTimePeriodInSecs = <number of seconds before data is rolled to frozen>


One thing to note here is that each bucket contains data across a given time span (e.g. it could be 1 hour or 1 day) up to a maximum of the value set for

maxHotSpanSecs 

  The earliest timestamp of an event (oldest event) and latest timestamp of an event (most recent) are in the name of the bucket (https://docs.splunk.com/Documentation/Splunk/9.0.3/Indexer/HowSplunkstoresindexes#Bucket_naming_conv...).

Before a bucket is rolled to frozen the timestamp of the most recent event must be older than the value set for frozenTimePeriodInSecs which means there can be cases where there is data that's older than you'd expect. 

 

 

0 Karma

zpasplunk
Explorer

Yes I was suspecting frozenTimePeriodInSecs as well based on the docs.. Thx

0 Karma
Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...