Deployment Architecture

Indexes.conf Stanza Storage for Indexer Cluster Settings Calculation

keithevanscdcr
Explorer

Hello all. Thanks in advance for your assistance.

I have a 6 node index cluster with a search factor of 6 and replication factor of 3. My ingest is 130GB per/day for proxy data
My Hardware: 7TB hot/warm SSD and 112TB cold 10k spindle per/indexer
I have a requirement for 30 days hot/warm and 1,065 days cold = 1,095 days (3 years) total

Calculations I have found say:
(Daily Avg. Ingest x Compression Rate x Num Days Retention) / # of Indexers

So:
Hot/Warm: 130GB * .6 compression * 15 days) / 6 = 390 GB p/indexer
Cold: 130GB * .6 compression * 1,065 days) / 6 = 13,845 GB p/indexer
Total: 130GB * .6 compression * 1,095 days) / 6 = 14,235 GB p/indexer

Given that, I think my indexes.conf would have:
[idx_proxy]
homePath = volume:primary/idx_proxy/db
coldPath = volume:secondary/idx_proxy/colddb
thawedPath = $SPLUNK_DB/idx_proxy/thaweddb
maxTotalDataSizeMB = 14576640
maxDataSize = auto_high_volume
homePath.maxDataSizeMB = 399360
frozenTimePeriodInSecs = 94608000

The question: With replication factor, I'm thinking this answer is not complete. Can anyone help? Updated formula/calc?

This is current storage:
[splunk@splunkidx1 ~]$ du -sh /splunkData/*/idx_proxy/
8.0T /splunkData/cold/idx_proxy/
947G /splunkData/hot/idx_proxy/

[splunk@splunkidx2 ~]$ du -sh /splunkData/*/idx_proxy/
8.0T /splunkData/cold/idx_proxy/
826G /splunkData/hot/idx_proxy/

[splunk@splunkidx3 ~]$
7.9T /splunkData/cold/idx_proxy/
955G /splunkData/hot/idx_proxy/

[splunk@splunkidx4 ~]$ du -sh /splunkData/*/idx_proxy/
7.8T /splunkData/cold/idx_proxy/
952G /splunkData/hot/idx_proxy/

[splunk@splunkidx5 ~]$ du -sh /splunkData/*/idx_proxy/
8.0T /splunkData/cold/idx_proxy/
936G /splunkData/hot/idx_proxy/

[splunk@splunkidx6 ~]$ du -sh /splunkData/*/idx_proxy/
7.8T /splunkData/cold/idx_proxy/
911G /splunkData/hot/idx_proxy/

Labels (1)
0 Karma
Get Updates on the Splunk Community!

Fueling your curiosity with new Splunk ILT and eLearning courses

At Splunk Education, we’re driven by curiosity—both ours and yours! That’s why we’re committed to delivering ...

Splunk AI Assistant for SPL 1.1.0 | Now Personalized to Your Environment for Greater ...

Splunk AI Assistant for SPL has transformed how users interact with Splunk, making it easier than ever to ...

Unleash Unified Security and Observability with Splunk Cloud Platform

     Now Available on Microsoft AzureOn Demand Now Step boldly into the AI revolution with enhanced security ...