Deployment Architecture

Indexes.conf Stanza Storage for Indexer Cluster Settings Calculation

keithevanscdcr
Explorer

Hello all. Thanks in advance for your assistance.

I have a 6 node index cluster with a search factor of 6 and replication factor of 3. My ingest is 130GB per/day for proxy data
My Hardware: 7TB hot/warm SSD and 112TB cold 10k spindle per/indexer
I have a requirement for 30 days hot/warm and 1,065 days cold = 1,095 days (3 years) total

Calculations I have found say:
(Daily Avg. Ingest x Compression Rate x Num Days Retention) / # of Indexers

So:
Hot/Warm: 130GB * .6 compression * 15 days) / 6 = 390 GB p/indexer
Cold: 130GB * .6 compression * 1,065 days) / 6 = 13,845 GB p/indexer
Total: 130GB * .6 compression * 1,095 days) / 6 = 14,235 GB p/indexer

Given that, I think my indexes.conf would have:
[idx_proxy]
homePath = volume:primary/idx_proxy/db
coldPath = volume:secondary/idx_proxy/colddb
thawedPath = $SPLUNK_DB/idx_proxy/thaweddb
maxTotalDataSizeMB = 14576640
maxDataSize = auto_high_volume
homePath.maxDataSizeMB = 399360
frozenTimePeriodInSecs = 94608000

The question: With replication factor, I'm thinking this answer is not complete. Can anyone help? Updated formula/calc?

This is current storage:
[splunk@splunkidx1 ~]$ du -sh /splunkData/*/idx_proxy/
8.0T /splunkData/cold/idx_proxy/
947G /splunkData/hot/idx_proxy/

[splunk@splunkidx2 ~]$ du -sh /splunkData/*/idx_proxy/
8.0T /splunkData/cold/idx_proxy/
826G /splunkData/hot/idx_proxy/

[splunk@splunkidx3 ~]$
7.9T /splunkData/cold/idx_proxy/
955G /splunkData/hot/idx_proxy/

[splunk@splunkidx4 ~]$ du -sh /splunkData/*/idx_proxy/
7.8T /splunkData/cold/idx_proxy/
952G /splunkData/hot/idx_proxy/

[splunk@splunkidx5 ~]$ du -sh /splunkData/*/idx_proxy/
8.0T /splunkData/cold/idx_proxy/
936G /splunkData/hot/idx_proxy/

[splunk@splunkidx6 ~]$ du -sh /splunkData/*/idx_proxy/
7.8T /splunkData/cold/idx_proxy/
911G /splunkData/hot/idx_proxy/

Labels (1)
0 Karma
Get Updates on the Splunk Community!

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...

Reminder! Splunk Love Promo: $25 Visa Gift Card for Your Honest SOAR Review With ...

We recently launched our first Splunk Love Special, and it's gone phenomenally well, so we're doing it again, ...