What happens when I update maxTotalDataSizeMB in a live environment

[Robbie1194]

Hi guys,

We are running a multi site index cluster with 12 indexers (6 across 2 sites). Our goal is to limit the size of one of our indexes.

We currently have an index that's sitting at 150gb and I was planning on using the maxTotalDataSizeMB to limit it to around 75gb.

The remaining 75gb should go to frozen, is this correct? (We don't actual freeze data so I assume it just gets deleted)

If i apply this update to maxTotalDataSizeMB when the cluster is up and running, will it just delete the data will carrying on with regular tasks as normal? As it is our production environment, we can't take the cluster down for even a minute. (rolling restarts are okay as it'll need one when I apply the new cluster bundle).

Does anyone know if its okay to apply this config while the cluster is up and running?

Cheers

[lfedak_splunk]

Hey @robbie1194, same as the last post! If this solution worked, please accept the answer. 🙂

[jtacy · Accepted solution]

Yes, this particular change may not even trigger a rolling restart when you apply the cluster bundle. The excess buckets will be frozen (deleted), starting with the oldest buckets. Just to clarify the behavior for other readers: maxTotalDataSizeMB applies to a specific index for each cluster member individually, not to the cluster as a whole. In this case with 12 indexers, setting a limit of 75 GB will result in total size of 900 GB across the cluster for the affected index.

This limit includes replicated buckets so cluster member failure or increasing replication factor could result in unexpected data loss. For this reason among others, I personally prefer to treat maxTotalDataSizeMB as a sort of emergency limit, using frozenTimePeriodInSecs as the primary tool to manage index size.

[ddrillic]

If you use Distribute Configuration Bundle from the master, you should be just fine.