https://community.splunk.com/t5/Getting-Data-In/Why-does-my-Splunk-indexer-keep-running-out-of-space-with-my/m-p/221236#M43433 <P>Problem solved thanks to Yasaswy and Rich.</P> <P>Reducing the maxWarmDBCount was the solution. </P> Thu, 25 Feb 2016 17:39:58 GMT gozulin 2016-02-25T17:39:58Z https://community.splunk.com/t5/Getting-Data-In/Why-does-my-Splunk-indexer-keep-running-out-of-space-with-my/m-p/221230#M43427 <P>The indexer pauses indexing when free space goes under 5GB on the main partition. This is caused by too many warm buckets filling up space instead of being moved to cold buckets on the larger, spinning-disk volume.</P> <P>I have a 3.2TB volume for hot/warm data on SSD and a 12TB volume for cold data on spinning disk. This is my indexes.conf. What am I doing wrong? </P> <PRE><CODE>#general maxWarmDBCount = 300 homePath.maxDataSizeMB = 3200000 coldPath.maxDataSizeMB = 12000000 #Volumes [volume:caliente] path = /splunkdata maxVolumeDataSizeMB = 3200000 [volume:frio] path = /cold maxVolumeDataSizeMB = 12000000 # indexes [_audit] thawedPath = $SPLUNK_DB/audit/thaweddb tstatsHomePath = volume:_splunk_summaries/audit/datamodel_summary homePath = volume:caliente/splunk_indexes/audit/db coldPath = volume:frio/_audit [shenanigans] thawedPath = $SPLUNK_DB/shenanigans/thaweddb tstatsHomePath = volume:_splunk_summaries/shenanigans/datamodel_summary maxConcurrentOptimizes = 6 maxHotIdleSecs = 86400 maxDataSize = auto_high_volume homePath = volume:caliente/splunk_indexes/shenanigans/db coldPath = volume:frio/shenanigans </CODE></PRE> Wed, 24 Feb 2016 21:04:20 GMT https://community.splunk.com/t5/Getting-Data-In/Why-does-my-Splunk-indexer-keep-running-out-of-space-with-my/m-p/221230#M43427 gozulin 2016-02-24T21:04:20Z https://community.splunk.com/t5/Getting-Data-In/Why-does-my-Splunk-indexer-keep-running-out-of-space-with-my/m-p/221231#M43428 <P>What error message or other symptom of a problem do you see?</P> Thu, 25 Feb 2016 12:40:07 GMT https://community.splunk.com/t5/Getting-Data-In/Why-does-my-Splunk-indexer-keep-running-out-of-space-with-my/m-p/221231#M43428 richgalloway 2016-02-25T12:40:07Z https://community.splunk.com/t5/Getting-Data-In/Why-does-my-Splunk-indexer-keep-running-out-of-space-with-my/m-p/221232#M43429 <P>The indexer pauses indexing when free space goes under 5GB on the main partition. This is caused by too many warm buckets filling up space instead of being moved to cold buckets on the larger, spinning-disk volume.</P> Thu, 25 Feb 2016 15:49:59 GMT https://community.splunk.com/t5/Getting-Data-In/Why-does-my-Splunk-indexer-keep-running-out-of-space-with-my/m-p/221232#M43429 gozulin 2016-02-25T15:49:59Z https://community.splunk.com/t5/Getting-Data-In/Why-does-my-Splunk-indexer-keep-running-out-of-space-with-my/m-p/221233#M43430 <P>Check out the maxHotSpanSecs attribute. The default value of 90 days may be too high for your environment.<BR /> Also, have a look at maxWarmDBCount.</P> Thu, 25 Feb 2016 16:14:23 GMT https://community.splunk.com/t5/Getting-Data-In/Why-does-my-Splunk-indexer-keep-running-out-of-space-with-my/m-p/221233#M43430 richgalloway 2016-02-25T16:14:23Z https://community.splunk.com/t5/Getting-Data-In/Why-does-my-Splunk-indexer-keep-running-out-of-space-with-my/m-p/221234#M43431 <P>Hi, Some observations... you have maxWarmDBCount = 300 and also have maxDataSize = auto_high_volume. On a 64bit system it would mean each bucket might take upto 10 GB and you have set the warm count at 300 (add 10 hot buckets as well). </P> <P>It looks like, depending on the activity, in the worst case scenario the homepath will be almost full 3.2 TB (300 * 10 for your warmbuckets at 10 GB + 10 hot buckets). Why don't you reduce the warm count? Eg... setting the warmcount to 250 would mean 500 GB space left on your homepath ..... as they will start rolling over.</P> Tue, 29 Sep 2020 08:54:44 GMT https://community.splunk.com/t5/Getting-Data-In/Why-does-my-Splunk-indexer-keep-running-out-of-space-with-my/m-p/221234#M43431 Yasaswy 2020-09-29T08:54:44Z https://community.splunk.com/t5/Getting-Data-In/Why-does-my-Splunk-indexer-keep-running-out-of-space-with-my/m-p/221235#M43432 <P>maxWarmDBCount is very effective. Thanks! Dialing it down to 50 should do the trick. Thanks.</P> <P>I don't understand how maxHotSpanSecs would help, I have 8 indexes and the fastest-growing ones roll over at around 6GB. Could you explain?</P> Thu, 25 Feb 2016 17:37:53 GMT https://community.splunk.com/t5/Getting-Data-In/Why-does-my-Splunk-indexer-keep-running-out-of-space-with-my/m-p/221235#M43432 gozulin 2016-02-25T17:37:53Z https://community.splunk.com/t5/Getting-Data-In/Why-does-my-Splunk-indexer-keep-running-out-of-space-with-my/m-p/221236#M43433 <P>Problem solved thanks to Yasaswy and Rich.</P> <P>Reducing the maxWarmDBCount was the solution. </P> Thu, 25 Feb 2016 17:39:58 GMT https://community.splunk.com/t5/Getting-Data-In/Why-does-my-Splunk-indexer-keep-running-out-of-space-with-my/m-p/221236#M43433 gozulin 2016-02-25T17:39:58Z