Deployment Architecture

Help with Smartstore index data retention

dm1
Communicator

I am struggling to understand data retention for Smartstore index. I guess I am thinking more from a non-smartstore way of data retention.

 

Below is my requirement

Data ingestion per day - 1.5 TB per day

Total retention - 90 days

Local storage - 50 days

Smartstore storage - 40 days

Multi-site indexer cluster with 2 indexers in each AWS AZ with RF - 3 and SF - 3

From the docs, I understand following parameters are key for retention

 

 

maxGlobalDataSizeMB
maxGlobalRawDataSizeMB
frozenTimePeriodInSecs

 

 

Since both maxGlobal* parameters apply to mainly warm buckets, how can I control data retention for each index as a total and not specifically to warm buckets, similar to maxTotalDataSizeMb ?

With non-smartstore index, this was so easy, you either just needed maxTotalDataSizeMb and frozenTimePeriodInSecs

Also, how can I control size of each index occupies on local storage (which is basically ability to configure something similar to homePath.maxDataSizeMB, which is again ignored for smartstore indexes)?

 

I am surprised all this is not documented clearly anywhere in official Splunk docs but some info can be found in forums.

How can I translate my requirements into these parameters on a per index basis ? Please help

 

Tags (1)
0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!