Splunk Enterprise

Indexes.conf Storage Configuration

jlc00
Loves-to-Learn Lots

I am in the middle of configuring a standalone Splunk installation.  I am getting confused about the different attributes that can be set for overall storage and per index.  It is a very small installation with only about 30 assets connected and about 2.3TB of storage to store data for a year.  I have the following configuration so far:

frozenTimePeriodinSecs = 31536000 #365 days

[volume:hotwarm]
path = <directory to hotwarm location>
maxVolumeDataSizeMB = 178176 #174GB

[volume:cold]
path = <directory to cold location>
maxVolumeDataSizeMB = 1970176 #1924GB

[network]
homePath = volume:hotwarm/network/db
coldPath = volume:cold/network/colddb
thawedPath = $SPLUNK_DB/network/thaweddb

[windows]
homePath = volume:hotwarm/windows/db
coldPath = volume:cold/windows/colddb
thawedPath = $SPLUNK_DB/windows/thaweddb

I'm not sure how to use the maxTotalDataSizeMB with maxVolumeDataSizeMB to keep from maxTotalDataSizeMB triggering a roll to frozen before the 365 days is up.  We currently do not have any idea how much data will be coming in.  Is it good practice to set maxTotalDataSizeMB for each index to the same size as maxVolumeDataSizeMB? I have seen this practice before...   And if so, is it the maxVolumeDataSize of the cold storage, or hot/warm/cold storage combined?

Labels (2)
0 Karma

Stefanie
Builder

I think you'll need to monitor your data for at least a month to see how much you're getting in, in order to fine tune your settings. 

Personally, I wouldn't set maxTotalDataSizeMB for each index to equal the same size as the maxVolumeDataSizeMB. If it's set the same, what happens in the event one of your indexes fills up that maxVolumeDataSizeMB faster than your other indexes? It won't leave enough room for your other indexes.

I think setting only the maxVolumeDataSizeMB will be fine until you monitor your ingestion to see which indexes are your busiest, and fine tune your settings from there.

 

 

 

richgalloway
SplunkTrust
SplunkTrust

maxTotalDataSizeMB sets the maximum size of an individual index whereas maxVolumeDataSizeMB sets the maximum size of all indexes sharing that volume.

The maximum size of an index applies to hot/warm as well as cold.  However, if hot/warm and cold are on different volumes then the max index size should not exceed the larger volume size.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...