Splunk Enterprise

Indexes.conf Storage Configuration

jlc00
Loves-to-Learn Lots

I am in the middle of configuring a standalone Splunk installation.  I am getting confused about the different attributes that can be set for overall storage and per index.  It is a very small installation with only about 30 assets connected and about 2.3TB of storage to store data for a year.  I have the following configuration so far:

frozenTimePeriodinSecs = 31536000 #365 days

[volume:hotwarm]
path = <directory to hotwarm location>
maxVolumeDataSizeMB = 178176 #174GB

[volume:cold]
path = <directory to cold location>
maxVolumeDataSizeMB = 1970176 #1924GB

[network]
homePath = volume:hotwarm/network/db
coldPath = volume:cold/network/colddb
thawedPath = $SPLUNK_DB/network/thaweddb

[windows]
homePath = volume:hotwarm/windows/db
coldPath = volume:cold/windows/colddb
thawedPath = $SPLUNK_DB/windows/thaweddb

I'm not sure how to use the maxTotalDataSizeMB with maxVolumeDataSizeMB to keep from maxTotalDataSizeMB triggering a roll to frozen before the 365 days is up.  We currently do not have any idea how much data will be coming in.  Is it good practice to set maxTotalDataSizeMB for each index to the same size as maxVolumeDataSizeMB? I have seen this practice before...   And if so, is it the maxVolumeDataSize of the cold storage, or hot/warm/cold storage combined?

Labels (2)
0 Karma

Stefanie
Builder

I think you'll need to monitor your data for at least a month to see how much you're getting in, in order to fine tune your settings. 

Personally, I wouldn't set maxTotalDataSizeMB for each index to equal the same size as the maxVolumeDataSizeMB. If it's set the same, what happens in the event one of your indexes fills up that maxVolumeDataSizeMB faster than your other indexes? It won't leave enough room for your other indexes.

I think setting only the maxVolumeDataSizeMB will be fine until you monitor your ingestion to see which indexes are your busiest, and fine tune your settings from there.

 

 

 

richgalloway
SplunkTrust
SplunkTrust

maxTotalDataSizeMB sets the maximum size of an individual index whereas maxVolumeDataSizeMB sets the maximum size of all indexes sharing that volume.

The maximum size of an index applies to hot/warm as well as cold.  However, if hot/warm and cold are on different volumes then the max index size should not exceed the larger volume size.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...