Deployment Architecture

What are the consequences with reducing the default value of maxTotalDataSizeMB?

ktang
Explorer

I am testing out a storage solution for frozen data. I don't have a large splunk roll out while testing so I want to get data into my frozen bucket as soon as possible. Can I safely reduce the default value (500,00MB) in maxTotalDataSizeMB to get data from Cold into Frozen?
I know I have to reduce other attributes to (e.g. maxWarmDBCount) too. Splunk Docs list an upper limit of setting the value. I would imagine seeing more writes to Frozen but I don't understand what the consequences potentially are.

The docs in indexes.conf also state that the attribute maxDataSize manages the size of all buckets but maxTotalDataSizeMB affects the data roll from cold to frozen. Is there a conflict if maxDataSize and maxTotalDataSizeMB are different values? Does one attribute take precedence over the other when it come to the cold bucket size?

Thank You very much.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

The index's data retention (when you want your cold bucket to roll to frozen bucket) is decided by two factors,
1) Maximum size of the index : defined by maxTotalDataSizeMB (total size of all hot/warm/cold buckets). If the total size of the index on disk exceeds to size (default 500GB) set in maxTotalDataSizeMB, the oldest cold bucket will roll to frozen.
2) Maximum age of bucket/data: defined by frozenTimePeriodInSecs (age of newest data in a bucket). If the timestamp of latest data in a cold bucket (every bucket will store data for a range of timestamp, latest and oldest) is older than the time period set by frozeTimePeriodInSecs, that bucket will roll over to frozen.

If you want to move your data to frozen sooner, decide which option would be betters, rollover based on total Size or age of data. (or use combination of both).

A good read on data retention here:
http://docs.splunk.com/Documentation/Splunk/6.4.1/Indexer/Setaretirementandarchivingpolicy
https://wiki.splunk.com/Deploy:BucketRotationAndRetention

The property maxDataSize defines the maximum size of hot bucket before it roll over to warm bucket, so it's the size of a bucket, not the index. (you may want to read the indexes.conf documentation again for these two properties maxDataSize and maxTotalDataSizeMB).

Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...