How to tune index retention?

[andynewsoncap]

Hello, as far as I can understand and please correct me if I am wrong. How an index behaves is based on it’s conf.

We have 5 IDX cluster and over 300 IDXes at this stage.

Our AIM is to keep 9 days (or there about) in Hot/Warm and 31days (or there about in Cold) One day maybe keep data in Frozen (but we are not there yet).

So as far as I understand in Splunk this is controlled by working out the Size per day of the data.  Then Take that number and x by 9 and x by 31.

To create

homePath.maxDataSizeMB =

And

coldPath.maxDataSizeMB =

And

maxTotalDataSizeMB =

Then finally

frozenTimePeriodInSecs =

To further complicate matters there is a compress ration for RAW and the compression ratio for TSIDX and the number of Indexers in the cluster and the replication factor.

All of which make for some fun and complicate calculation.

My question is this.

Have anyone come up with a way to do this?

Or at has someone worked out how to extract a list of IDX per INDEX Cluster and the current daily Data Rate.  And doing the same to extract per IDX per INDEX Cluster the compression Ratio of the RAX Data and the TSIDX

Then maybe its possible to do some magic in Excel to build out

Per IDX

homePath.maxDataSizeMB =

coldPath.maxDataSizeMB =

maxTotalDataSizeMB =

Thanks.

[andynewsoncap]

Thanks.  We can't be the only people to ask this.  Oh and we designed it with Splunk PS.  And there are very good reasons why we have so many indexes. 5 Index Cluster so spread-out so 70-100 per cluster.

I worked out I would have to take RF and SF into account. 

I was hoping someone may know how to pull size and compression factor from an SPL.

If not next stop Splunk on-demand i guess.

Thanks.  

[gcusello]

Hi @andynewsoncap,

let me understand: you have 5 Clusters of Indexers or 5 Indexers in one Cluster?

Anyway, also 70 indexes are very many in my opinion!

About Compression Factor the values are the ones I described: 0.15 for row data and 0.35 for indexes.

About Replication Factor and Search Factor, they depends on the affidability you want:

if you have 5 Indexers, how many Indexers can be down without missing data?

Replication Factor and search factor can be eual to number of indexers, in this case you have all the data in all indexers and you system have a consistent base of data also with 4 Indexers down but with a greater cost for storage, so you have to define the leverage of cost and affidability.

You can surely ask to Splunk PS on demand and I hint this or to ask to your reference Splunk Partner that should have a Splunk Architect.

Ciao.

Giuseppe

[andynewsoncap]

In fact we have 9 Index Clusters.

2 in US. 5 node Cluster and 3 node cluster

2 in APAC. 4 node Cluster and 3 node cluster

2 in LATAM. 4 node Cluster and 3 node cluster

2 in EMEA. 5 node Cluster and 3 node cluster

All reporting to one 9 node Search head cluster.  Which has its own 4 node ITSI Indexing Cluster.

2+ 2 + 2 + 2 + 1 

There are about 40,000 devices feeding it Data. lets say 10k / 10k / 10k / 10k 

All are set to 2 / 2. SF / RF.

Each Index Cluster has its own set of Indexes / Index names based on the Data it is collecting for that region.  So I can tell if this is EMEA data or USA Data. etc.etc.  Based of the source of the Indexer.  The customer has a habit of spinning up server in one region but colleting data for another region.  Yer dont ask.  Customer is always right. Right 🙂  For complex reason i will not go into here.  But this is how myself and Splunk PS designed it 3 years ago.  And its been running for 3 years now.

Now we need to ensure we are keeping the data for an appropriator amount of time.  Which we are not.

[gcusello]

Hi @andynewsoncap,

for the complexity of your architecture, this isn't a question for the Community, but it requires a deep analysis from a Splunk PS or a Certified Splunk Architect.

Ciao.

Giuseppe

[andynewsoncap]

Yes complexity of your architecture.  But not if you break it down.  

Forget about all my Cluster.

Simple question.

How can I find out the Daily Size and Compression Ratio for an Index.  I know you said its  (0.15 x RF + 0.35 x SF) but i not sure that's is right.  But it may well work given I need to allow fudge factor.

Then expand the question.  To all the Indexers on a given set of Indexers.  

Something like.

What is the size of each index on a given set of Indexer for the last 30 days.  Them Simply take that answer / 30 to give an approximate daily ingestion size in MB.  Then take the  (0.15 x RF + 0.35 x SF) to work up a number.  And times that but a number of days for Hot/Warm and a number of Days for Cold. i.e. 

homePath.maxDataSizeMB =

And

coldPath.maxDataSizeMB =

And

maxTotalDataSizeMB =

[gcusello]

Hi @andynewsoncap,

you can find the daily size of an index using the License Consuption Report .

the compression factor is usually around 50%, but a more detailed calculation is the one I shared, from the Splunk Architect Course.

total_disk_space = daily_indexing * (0.15 * RF + 0.35 * SF) * retention

obviously, this is the total required storage for a cluster, to have the storage required for each Indexer of a cluster you have to divide by the number of Indexers of the Cluster.

Ciao.

Giuseppe

[gcusello]

Hi @andynewsoncap,

it's very difficoult give some values for your requirements.

The first thing I have in mind is that probably you have too many indexes to manage: usually indexes are divided in different ones based on retention and access rights, what's te algorithm ofd you index division?

then the max dimension depends on the number and dimension of events: you could try an empirical approach extrapolating this dimension from the used disk space for a known number of events for each index and extrapolate them: 

at first extrapolating: e.g. if 1million of events uses 100 MB for an index in 1 day you have to calculate

hot_warm_dimension = 1_day_volume x hot_wart_retention x (0.15 x RF + 0.35 x SF)

cold_dimension = 1_day_volume x cold_retention x (0.15 x RF + 0.35 x SF)

anyway, consider alway a contingency or 20% on this space and monitor the real occupation.

Then you have to consider index replication that depends on the Replication Factor and on the Search Factor.

In conclusion, you can try a calculation but to be more sure I hint to engage a Splunk PS or at least a Splunk Certified Architect, this isn't a questin for the Community!

Ciao.

Giuseppe