Indexes.conf Storage Configuration

[jlc00]

I am in the middle of configuring a standalone Splunk installation.  I am getting confused about the different attributes that can be set for overall storage and per index.  It is a very small installation with only about 30 assets connected and about 2.3TB of storage to store data for a year.  I have the following configuration so far:

frozenTimePeriodinSecs = 31536000 #365 days

[volume:hotwarm]
path = <directory to hotwarm location>
maxVolumeDataSizeMB = 178176 #174GB

[volume:cold]
path = <directory to cold location>
maxVolumeDataSizeMB = 1970176 #1924GB

[network]
homePath = volume:hotwarm/network/db
coldPath = volume:cold/network/colddb
thawedPath = $SPLUNK_DB/network/thaweddb

[windows]
homePath = volume:hotwarm/windows/db
coldPath = volume:cold/windows/colddb
thawedPath = $SPLUNK_DB/windows/thaweddb

I'm not sure how to use the maxTotalDataSizeMB with maxVolumeDataSizeMB to keep from maxTotalDataSizeMB triggering a roll to frozen before the 365 days is up.  We currently do not have any idea how much data will be coming in.  Is it good practice to set maxTotalDataSizeMB for each index to the same size as maxVolumeDataSizeMB? I have seen this practice before...   And if so, is it the maxVolumeDataSize of the cold storage, or hot/warm/cold storage combined?

[Stefanie]

I think you'll need to monitor your data for at least a month to see how much you're getting in, in order to fine tune your settings. 

Personally, I wouldn't set maxTotalDataSizeMB for each index to equal the same size as the maxVolumeDataSizeMB. If it's set the same, what happens in the event one of your indexes fills up that maxVolumeDataSizeMB faster than your other indexes? It won't leave enough room for your other indexes.

I think setting only the maxVolumeDataSizeMB will be fine until you monitor your ingestion to see which indexes are your busiest, and fine tune your settings from there.

[richgalloway]

maxTotalDataSizeMB sets the maximum size of an individual index whereas maxVolumeDataSizeMB sets the maximum size of all indexes sharing that volume.

The maximum size of an index applies to hot/warm as well as cold.  However, if hot/warm and cold are on different volumes then the max index size should not exceed the larger volume size.