Deployment Architecture

Index Rotation & Backup Calculator

ruiaires
Path Finder

Hi all,

I've developed an Index Calculator Excel (in Google Spreadsheet) to help manage a multiple Index system with several log rotation policies and calculate stuff like maxWarmDBCount, frozenTimePeriodInSecs and maxTotalDataSizeMB.

This calculator allows allocating space individually for each index, input the current daily usage and estimate when (number of days) the space will run out. A Required Date Span for warm buckets can also be defined to suggest a maxWarmCount setting.

It manages 2 separate storages, one for Hot+Warm and other for Cold buckets.

I think this might be useful to share here or in the Splunk Wiki...

My question is: do you think this tool is accurate and can be usefull for others to use ?

Any comments are appreciated 😉

You can see and copy the document in here: http://goo.gl/SQQiY

Thanks

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

This seems like a good idea (I haven't looked at it) but note that 4.2 is coming out soon and it will have several additional index parameters for setting and controlling indexing sizes, so you may want to take a look at the new docs and spec file to see about updating it.

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi ruiaires

At first glace it really looks nice. did you cross check it with the information from here http://www.splunk.com/wiki/Deploy:BucketRotationAndRetention

regards

ruiaires
Path Finder

The ideia is to play around with different scenarios, change values and keep them updated as you are monitoring REAL DATA (data usage peaks in the time span of a year are very common on some systems) so there is a need to constantly review the calculations.... a ready to use excel does the trick 😉

0 Karma

ruiaires
Path Finder

Yes! I read ALL the documentation, wiki pages and answers in here.

It's all great stuff, but there is never any mention of multi-index calculations, specially when you have to prioritize and define "allocated" space in terms of the criticalness of each index and the "desired/required" timespan of available "warm" data (without going over the available disk!)

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...