Getting Data In

Why does my Splunk indexer keep running out of space with my current indexes.conf?

gozulin
Communicator

The indexer pauses indexing when free space goes under 5GB on the main partition. This is caused by too many warm buckets filling up space instead of being moved to cold buckets on the larger, spinning-disk volume.

I have a 3.2TB volume for hot/warm data on SSD and a 12TB volume for cold data on spinning disk. This is my indexes.conf. What am I doing wrong?

#general
maxWarmDBCount = 300 
homePath.maxDataSizeMB = 3200000
coldPath.maxDataSizeMB = 12000000

#Volumes
[volume:caliente]
path = /splunkdata
maxVolumeDataSizeMB = 3200000

[volume:frio]
path = /cold
maxVolumeDataSizeMB = 12000000

# indexes
[_audit]
thawedPath = $SPLUNK_DB/audit/thaweddb
tstatsHomePath = volume:_splunk_summaries/audit/datamodel_summary
homePath = volume:caliente/splunk_indexes/audit/db
coldPath = volume:frio/_audit

[shenanigans]
thawedPath = $SPLUNK_DB/shenanigans/thaweddb
tstatsHomePath = volume:_splunk_summaries/shenanigans/datamodel_summary
maxConcurrentOptimizes = 6
maxHotIdleSecs = 86400
maxDataSize = auto_high_volume
homePath = volume:caliente/splunk_indexes/shenanigans/db
coldPath = volume:frio/shenanigans
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Check out the maxHotSpanSecs attribute. The default value of 90 days may be too high for your environment.
Also, have a look at maxWarmDBCount.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

Check out the maxHotSpanSecs attribute. The default value of 90 days may be too high for your environment.
Also, have a look at maxWarmDBCount.

---
If this reply helps you, Karma would be appreciated.

gozulin
Communicator

Problem solved thanks to Yasaswy and Rich.

Reducing the maxWarmDBCount was the solution.

0 Karma

gozulin
Communicator

maxWarmDBCount is very effective. Thanks! Dialing it down to 50 should do the trick. Thanks.

I don't understand how maxHotSpanSecs would help, I have 8 indexes and the fastest-growing ones roll over at around 6GB. Could you explain?

0 Karma

Yasaswy
Contributor

Hi, Some observations... you have maxWarmDBCount = 300 and also have maxDataSize = auto_high_volume. On a 64bit system it would mean each bucket might take upto 10 GB and you have set the warm count at 300 (add 10 hot buckets as well).

It looks like, depending on the activity, in the worst case scenario the homepath will be almost full 3.2 TB (300 * 10 for your warmbuckets at 10 GB + 10 hot buckets). Why don't you reduce the warm count? Eg... setting the warmcount to 250 would mean 500 GB space left on your homepath ..... as they will start rolling over.

richgalloway
SplunkTrust
SplunkTrust

What error message or other symptom of a problem do you see?

---
If this reply helps you, Karma would be appreciated.

gozulin
Communicator

The indexer pauses indexing when free space goes under 5GB on the main partition. This is caused by too many warm buckets filling up space instead of being moved to cold buckets on the larger, spinning-disk volume.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...