topic Re: Indexes bucket settings in Splunk Enterprise

Indexes bucket settings

Abilan1 — Mon, 14 Mar 2016 16:06:50 GMT

Hi ,

I see that "Max size (MB) of hot/warm/cold bucket = auto" under my one of index, I would like to know about this setting, like how long data will be in hot/warm/cold buckets? also If we have this setting for index, then frozenTimePeriodInSecs will not work?

Re: Indexes bucket settings

yannK — Mon, 14 Mar 2016 16:14:36 GMT

This is probably the setting "maxDataSize " for the bucket size (auto = 750MB/ bucket)l.
This is not the size of the index, but the size of the unit of storages (buckets)

see in the specifications : http://docs.splunk.com/Documentation/Splunk/latest/admin/Indexesconf

maxDataSize = <positive integer>|auto|auto_high_volume
* The maximum size in MB for a hot DB to reach before a roll to warm is triggered.
* Specifying "auto" or "auto_high_volume" will cause Splunk to autotune this parameter (recommended).
* You should use "auto_high_volume" for high-volume indexes (such as the main index); otherwise, use "auto".  A "high volume index" would typically be considered one that gets over 10GB of data per day.
* Defaults to "auto", which sets the size to 750MB.
* "auto_high_volume" sets the size to 10GB on 64-bit, and 1GB on 32-bit  systems.
* Although the maximum value you can set this is 1048576 MB, which   corresponds to 1 TB, a reasonable number ranges anywhere from 100 to    50000.  Before proceeding with any higher value, please seek approval of    Splunk Support.
* If you specify an invalid number or string, maxDataSize will be auto    tuned.
* NOTE: The maximum size of your warm buckets may slightly exceed    'maxDataSize', due to post-processing and timing issues with the rolling   policy.

if you are looking for the index size limits, look at "maxTotalDataSizeMB" and the volumes definitions.

Re: Indexes bucket settings

Abilan1 — Mon, 14 Mar 2016 16:18:15 GMT

Hi ,

It means hot bucket size is 750 MB right? And what about Warm and cold bucket size? And where I can find this settings?

Re: Indexes bucket settings

yannK — Mon, 14 Mar 2016 17:03:16 GMT

The hot buckets are read/write, the warm and cold are read only. Therefore when the buckets rotate
hot -> warm -> cold, their size stays the same, and is defined by the size when they were hot.

Re: Indexes bucket settings

Abilan1 — Mon, 14 Mar 2016 17:19:23 GMT

So it means all hot, warm, cold buckets will be only 750 MB? After 750 MB filled in Cold db what will happen? data will be deleted?

Re: Indexes bucket settings

yannK — Mon, 14 Mar 2016 17:49:09 GMT

No, the bucket is size is not the index size, a bucket is an unit of storage time based, with span, number and size limits.
An index is composed of many buckets in different states.
they are also additional limits on the number of hot buckets, and warm buckets, but not cold buckets, for storage reasons.

example :
if for your index "myindex" maxTotalDataSizeMB = 100GB
then you will have inside buckets of chunk of data between a few MB to 750MB
to if you have a very streamline ingestion (and buckets of 750MB all) about 136 buckets.

(100*1024/750)