Deployment Architecture

Help with Smartstore index data retention

dm1
Contributor

I am struggling to understand data retention for Smartstore index. I guess I am thinking more from a non-smartstore way of data retention.

 

Below is my requirement

Data ingestion per day - 1.5 TB per day

Total retention - 90 days

Local storage - 50 days

Smartstore storage - 40 days

Multi-site indexer cluster with 2 indexers in each AWS AZ with RF - 3 and SF - 3

From the docs, I understand following parameters are key for retention

 

 

maxGlobalDataSizeMB
maxGlobalRawDataSizeMB
frozenTimePeriodInSecs

 

 

Since both maxGlobal* parameters apply to mainly warm buckets, how can I control data retention for each index as a total and not specifically to warm buckets, similar to maxTotalDataSizeMb ?

With non-smartstore index, this was so easy, you either just needed maxTotalDataSizeMb and frozenTimePeriodInSecs

Also, how can I control size of each index occupies on local storage (which is basically ability to configure something similar to homePath.maxDataSizeMB, which is again ignored for smartstore indexes)?

 

I am surprised all this is not documented clearly anywhere in official Splunk docs but some info can be found in forums.

How can I translate my requirements into these parameters on a per index basis ? Please help

 

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Index This | I’m short for "configuration file.” What am I?

May 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with a Special ...

New Articles from Academic Learning Partners, Help Expand Lantern’s Use Case Library, ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Your Guide to SPL2 at .conf24!

So, you’re headed to .conf24? You’re in for a good time. Las Vegas weather is just *chef’s kiss* beautiful in ...