topic Re: index bucket in Splunk Enterprise

index bucket

dall — Tue, 05 Jan 2021 05:24:48 GMT

[ind2]

HomePath = $SPLUNK_DB/ind2/db

ColdPath = $SPLUNK_DB/ind2/colddb

ThawedPath = $SPLUNK_DB/ind2/thaweddb

MaxHotBuckets = 10

MaxDataSize = 10000

MaxWarmDBCount = 300

MaxTotalDataSizeMB = 50000

FrozenTimePeriodInSecs = 31536000

ColdToFrozenDir = $SPLUNK_DB/ind2/frozendb

In this config how long data will reside in each bucket

as per my understanding after 10gb data in hot ll roll to warm and when 300 bucket in warm ll full that ll roll to cold

or 10 hot bucket each bucket in hot ll take 10gb data than it ll roll to warm

is there aany one who can clarify???

Re: index bucket

richgalloway — Tue, 05 Jan 2021 14:23:22 GMT

All we know for sure is buckets will live for at least 31,536,000 seconds (365 days). That's a minimum value because the life of a buckets is based on the age of the newest event in that bucket. Without knowing the ingestion rate it's impossible to know how long a bucket will remain before it is frozen. For the same reason, we can't know how long the bucket will remain at each tier.

Of course, once frozen the bucket will stay there forever or until you thaw or delete it.

Re: index bucket

scelikok — Tue, 05 Jan 2021 14:51:51 GMT

Hi @dall,

Addition to @richgalloway , since your low MaxTotalDataSizeMB setting, your warm db count cannot reach to 300. If there is no indexer restart 5 hot buckets will reach to MaxTotalDataSizeMB (50 GB) then go directly to frozen path (on every indexer restart, all hot buckets are rolled to warm). In order to keep data searchable for 365 days you should calculate the MaxTotalDataSizeMB using your daily ingestion rate.

You can check below documentation;

https://docs.splunk.com/Documentation/Splunk/8.1.1/Indexer/HowSplunkstoresindexes#How_data_ages

Re: index bucket

dall — Wed, 06 Jan 2021 05:39:00 GMT

maxwarmdbcount=300 means once 300 bucket ll full than roll to next bucket

from here we need to ser size of warm bucket ya how to know size of this warm bucket??

Re: index bucket

scelikok — Wed, 06 Jan 2021 05:56:04 GMT

maxwarmdbcount=300 means there can be maximum 300 warm buckets can exist on ind2 index HomePath.

Ideally all active bucket sizes can be maximum MaxDataSize which is 10GB on your ind2 index setting. Bucket sizes do not change while rolling to hot-->warm or warm-->cold. Only frozen buckets gets smaller because they contain only compressed raw-data.

Indexer restarts, wrong timestamp issues , low disk space may cause hot buckets to roll warm before reaching to MaxDataSize capacity. You can see many warm buckets smaller than MaxDataSize. For calculation you can consider them as MaxDataSize.

If this reply helps you an upvote is appreciated.