topic Re: Max Bucket Size warning in Deployment Architecture

Max Bucket Size warning

peter_gianusso — Fri, 25 Oct 2013 21:11:18 GMT

On the indexer in Splunk 6 getting this error message

10-25-2013 17:00:11.024 -0400 WARN IndexConfig - Max bucket size is larger than the index size limit. Please check your index configuration. idx=main; bucket size in MB (from maxDataSize) 10240, maxTotalDataSizeMB=1000

It does not state what index.

I checked the index configuration of the main application.

Max size (MB) of entire index : 50000
Max size (MB) of hot/warm/cold bucket is auto

Should i be worried about this message?

Re: Max Bucket Size warning

yannK — Mon, 28 Sep 2020 15:05:35 GMT

Yes, you can have a hot bucket larger than your maximum index size.
So the possible consequence is that a hot bucket will fill your storage, because it cannot be delete until it rotated to warm or cold.
So make sure than you have maxDataSize < maxTotalDataSizeMB

To confirm run a btool command on your indexes.conf

splunk cmd btool indexes list

and check for maxDataSize and maxTotalDataSizeMB

for the explanation of the indexes parameters :
see http://docs.splunk.com/Documentation/Splunk/latest/admin/Indexesconf

`
maxTotalDataSizeMB =
* The maximum size of an index (in MB).
* If an index grows larger than the maximum size, the oldest data is frozen.
* This paremeter only applies to hot, warm, and cold buckets. It does not apply to thawed buckets.
* Defaults to 500000.
* Highest legal value is 4294967295

maxDataSize = |auto|auto_high_volume
* The maximum size in MB for a hot DB to reach before a roll to warm is triggered.
* Specifying "auto" or "auto_high_volume" will cause Splunk to autotune this parameter (recommended).
* You should use "auto_high_volume" for high-volume indexes (such as the main
index); otherwise, use "auto". A "high volume index" would typically be
considered one that gets over 10GB of data per day.
* Defaults to "auto", which sets the size to 750MB.
* "auto_high_volume" sets the size to 10GB on 64-bit, and 1GB on 32-bit systems.
* Although the maximum value you can set this is 1048576 MB, which corresponds to 1 TB, a reasonable
number ranges anywhere from 100 to 50000. Before proceeding with any higher value, please seek
approval of Splunk Support.
* If you specify an invalid number or string, maxDataSize will be auto tuned.
* NOTE: The maximum size of your warm buckets may slightly exceed 'maxDataSize', due to post-processing and
timing issues with the rolling policy.

Re: Max Bucket Size warning

kristian_kolb — Fri, 25 Oct 2013 21:59:47 GMT

yes - it says that is in the main index (this is the default index).

you should probably look in either of these places

/opt/splunk/etc/apps//local/indexes.conf
/opt/splunk/etc/slave-apps//local/indexes.conf
/opt/splunk/etc/system/local/indexes.conf

note that for , you'll need to look in all apps in these directories.

The problem seems to be that you have configured a maximum size for the entire index to be 1GB, whereas the maximum size for any bucket within the index is at 10 GB. Thus - as soon as a hot bucket is rolled to warm, it will be frozen (most likely deleted).