Deployment Architecture

Index Rotation & Backup Calculator

ruiaires
Path Finder

Hi all,

I've developed an Index Calculator Excel (in Google Spreadsheet) to help manage a multiple Index system with several log rotation policies and calculate stuff like maxWarmDBCount, frozenTimePeriodInSecs and maxTotalDataSizeMB.

This calculator allows allocating space individually for each index, input the current daily usage and estimate when (number of days) the space will run out. A Required Date Span for warm buckets can also be defined to suggest a maxWarmCount setting.

It manages 2 separate storages, one for Hot+Warm and other for Cold buckets.

I think this might be useful to share here or in the Splunk Wiki...

My question is: do you think this tool is accurate and can be usefull for others to use ?

Any comments are appreciated 😉

You can see and copy the document in here: http://goo.gl/SQQiY

Thanks

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

This seems like a good idea (I haven't looked at it) but note that 4.2 is coming out soon and it will have several additional index parameters for setting and controlling indexing sizes, so you may want to take a look at the new docs and spec file to see about updating it.

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi ruiaires

At first glace it really looks nice. did you cross check it with the information from here http://www.splunk.com/wiki/Deploy:BucketRotationAndRetention

regards

ruiaires
Path Finder

The ideia is to play around with different scenarios, change values and keep them updated as you are monitoring REAL DATA (data usage peaks in the time span of a year are very common on some systems) so there is a need to constantly review the calculations.... a ready to use excel does the trick 😉

0 Karma

ruiaires
Path Finder

Yes! I read ALL the documentation, wiki pages and answers in here.

It's all great stuff, but there is never any mention of multi-index calculations, specially when you have to prioritize and define "allocated" space in terms of the criticalness of each index and the "desired/required" timespan of available "warm" data (without going over the available disk!)

0 Karma
Get Updates on the Splunk Community!

.conf23 Registration is Now Open!

Time to toss the .conf-etti 🎉 —  .conf23 registration is open!   Join us in Las Vegas July 17-20 for ...

Don't wait! Accept the Mission Possible: Splunk Adoption Challenge Now and Win ...

Attention everyone! We have exciting news to share! We are recruiting new members for the Mission Possible: ...

Unify Your SecOps with Splunk Mission Control

In today’s post, I'm excited to share some recent Splunk Mission Control innovations. With Splunk Mission ...