Getting Data In
Highlighted

Why does my Splunk indexer keep running out of space with my current indexes.conf?

Communicator

The indexer pauses indexing when free space goes under 5GB on the main partition. This is caused by too many warm buckets filling up space instead of being moved to cold buckets on the larger, spinning-disk volume.

I have a 3.2TB volume for hot/warm data on SSD and a 12TB volume for cold data on spinning disk. This is my indexes.conf. What am I doing wrong?

#general
maxWarmDBCount = 300 
homePath.maxDataSizeMB = 3200000
coldPath.maxDataSizeMB = 12000000

#Volumes
[volume:caliente]
path = /splunkdata
maxVolumeDataSizeMB = 3200000

[volume:frio]
path = /cold
maxVolumeDataSizeMB = 12000000

# indexes
[_audit]
thawedPath = $SPLUNK_DB/audit/thaweddb
tstatsHomePath = volume:_splunk_summaries/audit/datamodel_summary
homePath = volume:caliente/splunk_indexes/audit/db
coldPath = volume:frio/_audit

[shenanigans]
thawedPath = $SPLUNK_DB/shenanigans/thaweddb
tstatsHomePath = volume:_splunk_summaries/shenanigans/datamodel_summary
maxConcurrentOptimizes = 6
maxHotIdleSecs = 86400
maxDataSize = auto_high_volume
homePath = volume:caliente/splunk_indexes/shenanigans/db
coldPath = volume:frio/shenanigans
0 Karma
Highlighted

Re: Why does my Splunk indexer keep running out of space with my current indexes.conf?

SplunkTrust
SplunkTrust

What error message or other symptom of a problem do you see?

---
If this reply helps you, an upvote would be appreciated.
Highlighted

Re: Why does my Splunk indexer keep running out of space with my current indexes.conf?

Communicator

The indexer pauses indexing when free space goes under 5GB on the main partition. This is caused by too many warm buckets filling up space instead of being moved to cold buckets on the larger, spinning-disk volume.

0 Karma
Highlighted

Re: Why does my Splunk indexer keep running out of space with my current indexes.conf?

SplunkTrust
SplunkTrust

Check out the maxHotSpanSecs attribute. The default value of 90 days may be too high for your environment.
Also, have a look at maxWarmDBCount.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

Highlighted

Re: Why does my Splunk indexer keep running out of space with my current indexes.conf?

Contributor

Hi, Some observations... you have maxWarmDBCount = 300 and also have maxDataSize = autohighvolume. On a 64bit system it would mean each bucket might take upto 10 GB and you have set the warm count at 300 (add 10 hot buckets as well).

It looks like, depending on the activity, in the worst case scenario the homepath will be almost full 3.2 TB (300 * 10 for your warmbuckets at 10 GB + 10 hot buckets). Why don't you reduce the warm count? Eg... setting the warmcount to 250 would mean 500 GB space left on your homepath ..... as they will start rolling over.

Highlighted

Re: Why does my Splunk indexer keep running out of space with my current indexes.conf?

Communicator

maxWarmDBCount is very effective. Thanks! Dialing it down to 50 should do the trick. Thanks.

I don't understand how maxHotSpanSecs would help, I have 8 indexes and the fastest-growing ones roll over at around 6GB. Could you explain?

0 Karma
Highlighted

Re: Why does my Splunk indexer keep running out of space with my current indexes.conf?

Communicator

Problem solved thanks to Yasaswy and Rich.

Reducing the maxWarmDBCount was the solution.

0 Karma