Why does my Splunk indexer keep running out of space with my current indexes.conf?

[gozulin]

The indexer pauses indexing when free space goes under 5GB on the main partition. This is caused by too many warm buckets filling up space instead of being moved to cold buckets on the larger, spinning-disk volume.

I have a 3.2TB volume for hot/warm data on SSD and a 12TB volume for cold data on spinning disk. This is my indexes.conf. What am I doing wrong?

#general
maxWarmDBCount = 300 
homePath.maxDataSizeMB = 3200000
coldPath.maxDataSizeMB = 12000000

#Volumes
[volume:caliente]
path = /splunkdata
maxVolumeDataSizeMB = 3200000

[volume:frio]
path = /cold
maxVolumeDataSizeMB = 12000000

# indexes
[_audit]
thawedPath = $SPLUNK_DB/audit/thaweddb
tstatsHomePath = volume:_splunk_summaries/audit/datamodel_summary
homePath = volume:caliente/splunk_indexes/audit/db
coldPath = volume:frio/_audit

[shenanigans]
thawedPath = $SPLUNK_DB/shenanigans/thaweddb
tstatsHomePath = volume:_splunk_summaries/shenanigans/datamodel_summary
maxConcurrentOptimizes = 6
maxHotIdleSecs = 86400
maxDataSize = auto_high_volume
homePath = volume:caliente/splunk_indexes/shenanigans/db
coldPath = volume:frio/shenanigans

[richgalloway · Accepted solution]

Check out the maxHotSpanSecs attribute. The default value of 90 days may be too high for your environment.
Also, have a look at maxWarmDBCount.

[gozulin]

Problem solved thanks to Yasaswy and Rich.

Reducing the maxWarmDBCount was the solution.

[gozulin]

maxWarmDBCount is very effective. Thanks! Dialing it down to 50 should do the trick. Thanks.

I don't understand how maxHotSpanSecs would help, I have 8 indexes and the fastest-growing ones roll over at around 6GB. Could you explain?

[Yasaswy]

Hi, Some observations... you have maxWarmDBCount = 300 and also have maxDataSize = auto_high_volume. On a 64bit system it would mean each bucket might take upto 10 GB and you have set the warm count at 300 (add 10 hot buckets as well).

It looks like, depending on the activity, in the worst case scenario the homepath will be almost full 3.2 TB (300 * 10 for your warmbuckets at 10 GB + 10 hot buckets). Why don't you reduce the warm count? Eg... setting the warmcount to 250 would mean 500 GB space left on your homepath ..... as they will start rolling over.

[richgalloway]

What error message or other symptom of a problem do you see?

[gozulin]

The indexer pauses indexing when free space goes under 5GB on the main partition. This is caused by too many warm buckets filling up space instead of being moved to cold buckets on the larger, spinning-disk volume.