Why are we not seeing expected behavior for maxTotalDataSizeMB in indexes.conf?

[Admiral_Marith]

I'm seeing a sudden spike in data coming from our firewalls (edge and internal). On average an increase of 202% daily. It's caused a 42% surge in my license use. While we're chasing that down, I wanted to make sure that I'm not shipping things to frozen prematurely.

That said: Indexes.conf snippet for networks

[networks]
homePath   = volume:hot/networks/db
coldPath   = volume:cold/networks/colddb
thawedPath = $SPLUNK_DB/networks/thaweddb
maxTotalDataSizeMB = 2541818
homePath.maxDataSizeMB = 1694630
coldPath.maxDataSizeMB = 847188
#explicit path to frozen directory
coldToFrozenDir = /splunkdatafrozen/networks

So I would expect a total footprint of hot/warm/cold to be 2.54 TB.

My actual footprint seems to be....

:/splunkdatahot # du -hs networks/
213G    networks/
:/splunkdatacold # du -hs networks/
828G    networks/

For a total of 1041 GB.

What's frosting my cookies the wrong flavor is the face that my homePath.maxDataSizeMB is set to 1.694 TB, but hot/warm only has 213G, whereas coldPath.maxDataSizeMB is 847 GB. Cold use appears to be close to that, but Hot/Warm isn't close to that and the footprint is NOT increasing in hot/warm day to day.

So what is it in indexes.conf config for this index we are doing wrong?

Please note that while we're sorting out where we're actually going to put frozen, my hot/warm is larger than cold, so we had been trying to shoot for around a 70/30 split between hot/cold. (I know that's inverted but I have a large amount of SSD here - whereas we don't have SAN for cold or frozen - yet)

Any insight appreciated.

-The Admiral.

[lguinn2 · Accepted solution]

I think this may be your answer, from indexes.conf.spec

maxWarmDBCount = <nonnegative integer>
* The maximum number of warm buckets.
* Warm buckets are located in the <homePath> for the index.
* If set to zero, Splunk will not retain any warm buckets
  (will roll them to cold as soon as it can)
* Highest legal value is 4294967295
* Defaults to 300.

So when your warm bucket count hits 301, the oldest warm bucket is moved to cold - regardless of how much space you have.

Also the maximum size parameter only applies to hot, warm and cold buckets. Frozen and thawed buckets do not count, and Splunk will not remove them.

[Admiral_Marith]

So if I understand this correctly. that limit is on a per index basis, so if one sets it globally in [default] for the indexes.conf to say 1200, that's 1200 warm buckets per index.

[lguinn2]

Yes, this limit is per index. You can set it globally, or you can set it for each index differently.