Splunk Enterprise

index bucket

dall
Path Finder

 [ind2]

HomePath = $SPLUNK_DB/ind2/db

ColdPath = $SPLUNK_DB/ind2/colddb

ThawedPath = $SPLUNK_DB/ind2/thaweddb

MaxHotBuckets = 10

MaxDataSize = 10000

MaxWarmDBCount = 300

MaxTotalDataSizeMB = 50000

FrozenTimePeriodInSecs = 31536000 

 ColdToFrozenDir = $SPLUNK_DB/ind2/frozendb

 

In this config how long data will reside in each bucket

as per my understanding after 10gb data in hot ll roll to warm and when 300 bucket in warm ll full that ll roll to cold 

or 10 hot bucket each bucket in hot ll take 10gb data than it ll roll to warm

is there aany one who can clarify??? 

Labels (1)
0 Karma

scelikok
Champion

maxwarmdbcount=300 means there can be maximum 300 warm buckets can exist on ind2 index HomePath.

Ideally all active bucket sizes can be maximum MaxDataSize which is 10GB on your ind2 index setting. Bucket sizes do not change while rolling to hot-->warm or warm-->cold. Only frozen buckets gets smaller because they contain only compressed raw-data.

Indexer restarts, wrong timestamp issues , low disk space may cause hot buckets to roll warm before reaching to MaxDataSize capacity. You can see many warm buckets smaller than MaxDataSize.  For calculation you can consider them as MaxDataSize.

 

If this reply helps you an upvote is appreciated.

If this reply helps you an upvote is appreciated.

scelikok
Champion

Hi @dall,

Addition to @richgalloway , since your low MaxTotalDataSizeMB setting, your warm db count cannot reach to 300. If there is no indexer restart 5 hot buckets will reach to MaxTotalDataSizeMB (50 GB) then go directly to frozen path (on every  indexer restart, all hot buckets are rolled to warm).  In order to keep data searchable for 365 days you should calculate the MaxTotalDataSizeMB  using your daily ingestion rate. 

You can check below documentation;

https://docs.splunk.com/Documentation/Splunk/8.1.1/Indexer/HowSplunkstoresindexes#How_data_ages 

 

If this reply helps you an upvote is appreciated.
0 Karma

dall
Path Finder

maxwarmdbcount=300 means once 300 bucket ll full than roll to next bucket

from here we need to ser size of warm bucket ya how to know size of this warm bucket??

0 Karma

richgalloway
SplunkTrust
SplunkTrust

All we know for sure is buckets will live for at least 31,536,000 seconds (365 days).  That's a minimum value because the life of a buckets is based on the age of the newest event in that bucket.  Without knowing the ingestion rate it's impossible to know how long a bucket will remain before it is frozen.  For the same reason, we can't know how long the bucket will remain at each tier.

Of course, once frozen the bucket will stay there forever or until you thaw or delete it.

---
If this reply helps you, an upvote would be appreciated.
0 Karma