topic Re: What happens when maxVolumeDataSizeMB parameters will be decreased? in Deployment Architecture

What happens when maxVolumeDataSizeMB parameters will be decreased?

mkelderm — Mon, 03 Feb 2014 19:01:44 GMT

Hi,

Our maxVolumeDataSizeMB parameter for the warm-bucketss is set to 1.9 TB. What happens if I set this parameters to 1.5 TB? Will the warm-buckets moved to cold after the restart?

Greetz,

Marc

Re: What happens when maxVolumeDataSizeMB parameters will be decreased?

somesoni2 — Mon, 03 Feb 2014 19:05:11 GMT

See this
http://answers.splunk.com/answers/45319/does-maxvolumedatasizemb-in-indexesconf-do-anything

Re: What happens when maxVolumeDataSizeMB parameters will be decreased?

yannK — Mon, 03 Feb 2014 19:15:04 GMT

The non hot buckets containing the oldest events will be frozen to meet the requirement.

Will the warm-buckets moved to cold after the restart

NO, they will be frozen (i.e. deleted if no frozen location or script is defined) not moved from warm to cold.

Re: What happens when maxVolumeDataSizeMB parameters will be decreased?

mkelderm — Mon, 03 Feb 2014 19:46:39 GMT

Can I find out wich buckets are frozen and move them to cold?

Re: What happens when maxVolumeDataSizeMB parameters will be decreased?

kristian_kolb — Mon, 03 Feb 2014 20:08:36 GMT

You need to define a frozen path, so that splunk knows where to put them (and not delete them). However, once they are frozen, they are no longer searchable, as only the raw data, and not the tsidx files are retained.

You can figure out which of the buckets that are likely to be frozen.

This will require that you find the size reduction / bucket size number of buckets and move them manually. In your case that could be (400 GB / 10 GB) ~40 buckets, but you should probably take a few more. Take the 50 buckets with the lowest values of X, where X is the 1st epoch timestamp part of the dirname

Re: What happens when maxVolumeDataSizeMB parameters will be decreased?

kristian_kolb — Mon, 03 Feb 2014 20:09:54 GMT

oops. 2nd part of the dirname, but the 1st epoch. my bad. corrected above already.

Re: What happens when maxVolumeDataSizeMB parameters will be decreased?

kylekoza — Tue, 24 Mar 2015 22:25:14 GMT

This seems to contradict you.

When a volume containing warm buckets reaches its maxVolumeDataSizeMB, it starts rolling buckets to cold. When a volume containing cold buckets reaches its maxVolumeDataSizeMB, it starts rolling buckets to frozen. If a volume contains both warm and cold buckets (which will happen if an index's homePath and coldPath are both set to the same volume), the oldest bucket will be rolled to frozen.

http://docs.splunk.com/Documentation/Splunk/6.2.2/Indexer/Configureindexstoragesize#Configure_index_size_with_volumes

Re: What happens when maxVolumeDataSizeMB parameters will be decreased?

akira_splunk — Fri, 09 Jun 2017 20:03:14 GMT

I can say from recent experience in 2017 on Splunk 6.5 that the buckets will roll from warm to cold, even if the cold volume and warm volume are the same.

A customer maximized their disk space, and the indexer stopped writing to disk. They reduced the maxVolumeDataSizeMB setting on the cluster master, and redeployed the bundle to the indexers. Through the monitoring console, we can instantly see the warm bucket size and usage decrease, and the size of the cold bucket increase usage increase. We also validated by looking at the filesystem.

The old data was NOT frozen!

Re: What happens when maxVolumeDataSizeMB parameters will be decreased?

akira_splunk — Fri, 09 Jun 2017 20:15:13 GMT

I can say from recent experience in 2017 on Splunk 6.5 that the buckets will roll from warm to cold, even if the cold volume and warm volume are the same.

One of my customer had their warm maxVolumeDataSizeMB set to 1.5TB, and their cold maxVolumeDataSizeMB also set to 1.5TB. However, they only had 1.5TB of disk space available, TOTAL!

We reduced the maxVolumeDataSizeMB for both hot and cold to 725GB each. Upon deploying the bundle from the cluster master, through the monitoring console, we can instantly see the warm bucket size and usage decrease to 725GB, and the size of the cold bucket increase to 725GB. We also validated by looking at the filesystem on one of the indexers.

The old data was NOT frozen!