Monitoring Splunk

Search not executed: The minimum free disk space (50000MB) reached for /home/username/splunk/var/run/splunk/dispatch. user=admin.

splunkfly
New Member

Even After increasing the size in the settings-->general settings-->Pause indexing if free disk space (in MB) falls below-->50000MB.

It is still not resolved the issue. Does anyone have solution get rid of this issue.

0 Karma

ddrillic
Ultra Champion

We ended up having a cronjob which clears the older directories. Something like -

find /opt/splunk/var/run/splunk/dispatch -type d -mtime +30 | xargs rm -rf

Not sure if it's the optimal solution, but it saves us lots of headaches ; - )

0 Karma

lguinn2
Legend

That setting ("Pause Indexing") means "if there is not 'this much' free space left on disk, stop indexing until there is more free space." This setting is also used for searches. When Splunk runs a search, it needs disk space for saving the search results. The error message says "hey, I don't have enough free disk for saving the search results." The same setting is used for both.

Increasing this number is NOT what you want!

Search results are usually only temporary, but if you have a lot of people saving their search results, they could be using a lot of space.

Or maybe you just don't have enough disk space for your indexes AND your search results.

You can temporarily set the number lower - but you need to find a more permanent resolution to the problem.

Figure out how much indexed data you have on disk and see if you need (1) more disk or (2) to limit the size of the indexers.
Decide if you need to set disk quotas for users to limit the amount of disk they can use for search results. (But be careful, because you want them to still be able to run searches and dashboards!)

0 Karma

lguinn2
Legend

Well, actually you can permanently set the number lower. But you still need to figure out what is consuming the disk, because it is probably just going to keep growing until the problem occurs again.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Announcing Modern Navigation: A New Era of Splunk User Experience

We are excited to introduce the Modern Navigation feature in the Splunk Platform, available to both cloud and ...

Modernize your Splunk Apps – Introducing Python 3.13 in Splunk

We are excited to announce that the upcoming releases of Splunk Enterprise 10.2.x and Splunk Cloud Platform ...

Step into “Hunt the Insider: An Splunk ES Premier Mystery” to catch a cybercriminal ...

After a whole week of being on call, you fell asleep on your keyboard, and you hit a sequence of buttons that ...