Monitoring Splunk
Highlighted

Search not executed: The minimum free disk space (50000MB) reached for /home/username/splunk/var/run/splunk/dispatch. user=admin.

New Member

Even After increasing the size in the settings-->general settings-->Pause indexing if free disk space (in MB) falls below-->50000MB.

It is still not resolved the issue. Does anyone have solution get rid of this issue.

0 Karma
Highlighted

Re: Search not executed: The minimum free disk space (50000MB) reached for /home/username/splunk/var/run/splunk/dispatch. user=admin.

Legend

That setting ("Pause Indexing") means "if there is not 'this much' free space left on disk, stop indexing until there is more free space." This setting is also used for searches. When Splunk runs a search, it needs disk space for saving the search results. The error message says "hey, I don't have enough free disk for saving the search results." The same setting is used for both.

Increasing this number is NOT what you want!

Search results are usually only temporary, but if you have a lot of people saving their search results, they could be using a lot of space.

Or maybe you just don't have enough disk space for your indexes AND your search results.

You can temporarily set the number lower - but you need to find a more permanent resolution to the problem.

Figure out how much indexed data you have on disk and see if you need (1) more disk or (2) to limit the size of the indexers.
Decide if you need to set disk quotas for users to limit the amount of disk they can use for search results. (But be careful, because you want them to still be able to run searches and dashboards!)

0 Karma
Highlighted

Re: Search not executed: The minimum free disk space (50000MB) reached for /home/username/splunk/var/run/splunk/dispatch. user=admin.

Legend

Well, actually you can permanently set the number lower. But you still need to figure out what is consuming the disk, because it is probably just going to keep growing until the problem occurs again.

0 Karma
Highlighted

Re: Search not executed: The minimum free disk space (50000MB) reached for /home/username/splunk/var/run/splunk/dispatch. user=admin.

Ultra Champion

We ended up having a cronjob which clears the older directories. Something like -

find /opt/splunk/var/run/splunk/dispatch -type d -mtime +30 | xargs rm -rf

Not sure if it's the optimal solution, but it saves us lots of headaches ; - )

0 Karma