Deployment Architecture

Help with Smartstore index data retention

dm1
Contributor

I am struggling to understand data retention for Smartstore index. I guess I am thinking more from a non-smartstore way of data retention.

 

Below is my requirement

Data ingestion per day - 1.5 TB per day

Total retention - 90 days

Local storage - 50 days

Smartstore storage - 40 days

Multi-site indexer cluster with 2 indexers in each AWS AZ with RF - 3 and SF - 3

From the docs, I understand following parameters are key for retention

 

 

maxGlobalDataSizeMB
maxGlobalRawDataSizeMB
frozenTimePeriodInSecs

 

 

Since both maxGlobal* parameters apply to mainly warm buckets, how can I control data retention for each index as a total and not specifically to warm buckets, similar to maxTotalDataSizeMb ?

With non-smartstore index, this was so easy, you either just needed maxTotalDataSizeMb and frozenTimePeriodInSecs

Also, how can I control size of each index occupies on local storage (which is basically ability to configure something similar to homePath.maxDataSizeMB, which is again ignored for smartstore indexes)?

 

I am surprised all this is not documented clearly anywhere in official Splunk docs but some info can be found in forums.

How can I translate my requirements into these parameters on a per index basis ? Please help

 

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...