topic Re: Index Rotation & Backup Calculator in Deployment Architecture

Index Rotation & Backup Calculator

ruiaires — Thu, 03 Mar 2011 21:00:31 GMT

Hi all,

I've developed an Index Calculator Excel (in Google Spreadsheet) to help manage a multiple Index system with several log rotation policies and calculate stuff like maxWarmDBCount, frozenTimePeriodInSecs and maxTotalDataSizeMB.

This calculator allows allocating space individually for each index, input the current daily usage and estimate when (number of days) the space will run out. A Required Date Span for warm buckets can also be defined to suggest a maxWarmCount setting.

It manages 2 separate storages, one for Hot+Warm and other for Cold buckets.

I think this might be useful to share here or in the Splunk Wiki...

My question is: do you think this tool is accurate and can be usefull for others to use ?

Any comments are appreciated 😉

You can see and copy the document in here: http://goo.gl/SQQiY

Thanks

Re: Index Rotation & Backup Calculator

MuS — Thu, 03 Mar 2011 21:14:28 GMT

Hi ruiaires

At first glace it really looks nice. did you cross check it with the information from here http://www.splunk.com/wiki/Deploy:BucketRotationAndRetention

regards

Re: Index Rotation & Backup Calculator

ruiaires — Thu, 03 Mar 2011 21:57:15 GMT

Yes! I read ALL the documentation, wiki pages and answers in here.

It's all great stuff, but there is never any mention of multi-index calculations, specially when you have to prioritize and define "allocated" space in terms of the criticalness of each index and the "desired/required" timespan of available "warm" data (without going over the available disk!)

Re: Index Rotation & Backup Calculator

ruiaires — Thu, 03 Mar 2011 21:57:40 GMT

The ideia is to play around with different scenarios, change values and keep them updated as you are monitoring REAL DATA (data usage peaks in the time span of a year are very common on some systems) so there is a need to constantly review the calculations.... a ready to use excel does the trick 😉

Re: Index Rotation & Backup Calculator

gkanapathy — Sat, 05 Mar 2011 08:32:59 GMT

This seems like a good idea (I haven't looked at it) but note that 4.2 is coming out soon and it will have several additional index parameters for setting and controlling indexing sizes, so you may want to take a look at the new docs and spec file to see about updating it.