Deployment Architecture

Can "maxTotalDataSizeMB" & "frozenTimePeriodInSecs" be combined for Index config ?

pkumar9610
Explorer

HI Friends,

I am using below config for creating Indexes in both my QA & Production Cluster. At this point, I am only using retention period for Indexes but it is not helping in capacity management. Can I add frozenTimePeriodInSecs to this config so that, if it reaches capacity limit, it will take care of it ?

[ship]
homePath   = volume:primary/ship/db
coldPath   = volume:primary/ship/colddb
thawedPath = $SPLUNK_DB/ship/thaweddb
frozenTimePeriodInSecs=10368000

frozenTimePeriodInSecs when it reaches the capacity limit, does it remove the old logs and continue Indexing new logs just like frozenTimePeriodInSecs? Or does it just stop Indexing when it reaches the limit ?

Thanks,
-Prashanth

0 Karma
1 Solution

Rob2520
Communicator

"maxTotalDataSizeMB" takes precedence over other "frozenTimePeriodInSecs".

If the index grows beyond maxTotalDataSizeMB megabytes before frozenTimePeriodInSecs seconds have passed, data could prematurely roll to frozen, and if frozenTimePeriodInSecs comes first, then data will be rolled to frozen as well.

To answer your question: YES. If your raw data reaches "frozenTimePeriodInSecs" seconds, then you will start loosing old data and continue indexing new data.

View solution in original post

0 Karma

Rob2520
Communicator

"maxTotalDataSizeMB" takes precedence over other "frozenTimePeriodInSecs".

If the index grows beyond maxTotalDataSizeMB megabytes before frozenTimePeriodInSecs seconds have passed, data could prematurely roll to frozen, and if frozenTimePeriodInSecs comes first, then data will be rolled to frozen as well.

To answer your question: YES. If your raw data reaches "frozenTimePeriodInSecs" seconds, then you will start loosing old data and continue indexing new data.

0 Karma

sakthiganesht
New Member

What happens when the frozenTimePeriodInSecs is reached but maxTotalDataSizeMB is not reached? Will it freeze indexed data older than frozenTimePeriodInSecs or continue to store them in colddb till the size reaches maxTotalDataSizeMB ?

0 Karma

ddrillic
Ultra Champion

maxTotalDataSizeMB and frozenTimePeriodInSecs coexist ; -)

0 Karma

pkumar9610
Explorer

I have updated my config to have both frozenTimePeriodInSecs & maxTotalDataSizeMB, but I don't see it is limiting to 1024MB. IS some thing wrong with my config here ?

[sse-router-qa]
homePath = volume:primary/sse-router-qa/db
coldPath = volume:primary/sse-router-qa/colddb
thawedPath = $SPLUNK_DB/sse-router-qa/thaweddb
frozenTimePeriodInSecs=172800
maxTotalDataSizeMB = 1024
maxHotBuckets = 6

0 Karma

pkumar9610
Explorer

Thank you for the info.

Lets say for example if I have set maxTotalDataSizeMB=100GB, is this 100GB is the RAW data size or after Splunk does it compression.

IF it is RAW data, how much size will it be after compression ?
And do I need to do this capacity planning with the RAW data size coming ?

0 Karma

jtacy
Builder

maxTotalDataSizeMB is the maximum total size of all buckets associated with an index. This includes the indexes (tsidx files) and compressed raw data (journal.gz). It also includes the buckets replicated from other indexers in your cluster.

The compression ratio for raw data varies, but you'll probably find that the index portion of the bucket is generally larger than the compressed raw data portion. An index can vary dramatically in size relative to the raw data depending on the number of unique terms (segments) that Splunk needs to index. That's especially true if you're using any index-time field extractions. Splunk buckets are just collections of files so you can look at them to compare the ratio for your own data. Splunk's official documentation about this is at:
https://docs.splunk.com/Documentation/Splunk/7.1.3/Capacity/Estimateyourstoragerequirements

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...