Indexing stopped due to low disk space for one source

msarro — Wed, 06 Jun 2012 18:11:05 GMT

Hey everyone. I have several sources being spread over 4 indexers.
I periodically receive error messages stating that space is low on /splunk/hot/{some source name}, specifically "Search peer server-chi-a2.sys.mycompany.net has the following message: You are low in disk space on partition "/splunk/hot/AS-CDR". Indexing has been paused. Will resume when free disk space rises above 2000MB." It shows up for around 10 minutes, and then disappears.

Here is my indexes.conf file's contents:

[XS]
disabled=false
homePath=volume:HOT/XS-CDR
coldPath=volume:COLD/XS-CDR
thawedPath=/splunk/thawed/XS-CDR
maxDataSize=auto

[AS]
disabled=false
homePath=volume:HOT/AS-CDR
coldPath=volume:COLD/AS-CDR
thawedPath=/splunk/thawed/AS-CDR
maxDataSize=auto

[PBTS]
disabled=false
homePath=volume:HOT/PBTS
coldPath=volume:COLD/PBTS
thawedPath=/splunk/thawed/PBTS
maxDataSize=auto_high_volume

[CMS]
disabled=false
homePath=volume:HOT/CMS
coldPath=volume:COLD/CMS
thawedPath=/splunk/thawed/CMS
maxDataSize=auto_high_volume

[MSP]
disabled=false
homePath=volume:HOT/MSP
coldPath=volume:COLD/MSP
thawedPath=/splunk/thawed/MSP
maxDataSize=auto

[KPI]
disabled=false
homePath=volume:HOT/KPI
coldPath=volume:COLD/KPI
thawedPath=/splunk/thawed/KPI
maxDataSize=auto

[LICENSING]
disabled=false
homePath=volume:HOT/LICENSING
coldPath=volume:COLD/LICENSING
thawedPath=/splunk/thawed/LICENSING
maxDataSize=auto

[CER]
disabled=false
homePath=volume:HOT/CER
coldPath=volume:COLD/CER
thawedPath=/splunk/thawed/CER
maxDataSize=auto_high_volume

[volume:HOT]
path=/splunk/hot
maxVolumeDataSizeMB=140000

[volume:COLD]
path=/splunk/cold
maxVolumeDataSizeMB=840000

The /splunk/hot LUN is 150 GB, and the /splunk/cold LUN is ~850GB (both housed on a SAN). I used volume sizing for the configuration as I did because I don't truly know how much space each individual source will use as the devices are incredibly bursty, and this should let splunk control it for me. I also purposely left the volume sizes around 10GB lower than the LUN size. If splunk is listening to its config files, I should never see a <2000MB free error, since at all times I should have at least 10GB free on each LUN. Has anyone else seen this? Is there something wrong with my config that I'm missing? I'd appreciate the help. Thanks!

Re: Indexing stopped due to low disk space for one source

mad4wknds — Mon, 03 Mar 2014 01:57:15 GMT

I have only seen this where the indexer is also configured as a deployment server.

topic Indexing stopped due to low disk space for one source in Getting Data In

Indexing stopped due to low disk space for one source

Re: Indexing stopped due to low disk space for one source