Monitoring Splunk

Search not executed: The minimum free disk space (50000MB) reached for /home/username/splunk/var/run/splunk/dispatch. user=admin.

splunkfly
New Member

Even After increasing the size in the settings-->general settings-->Pause indexing if free disk space (in MB) falls below-->50000MB.

It is still not resolved the issue. Does anyone have solution get rid of this issue.

0 Karma

ddrillic
Ultra Champion

We ended up having a cronjob which clears the older directories. Something like -

find /opt/splunk/var/run/splunk/dispatch -type d -mtime +30 | xargs rm -rf

Not sure if it's the optimal solution, but it saves us lots of headaches ; - )

0 Karma

lguinn2
Legend

That setting ("Pause Indexing") means "if there is not 'this much' free space left on disk, stop indexing until there is more free space." This setting is also used for searches. When Splunk runs a search, it needs disk space for saving the search results. The error message says "hey, I don't have enough free disk for saving the search results." The same setting is used for both.

Increasing this number is NOT what you want!

Search results are usually only temporary, but if you have a lot of people saving their search results, they could be using a lot of space.

Or maybe you just don't have enough disk space for your indexes AND your search results.

You can temporarily set the number lower - but you need to find a more permanent resolution to the problem.

Figure out how much indexed data you have on disk and see if you need (1) more disk or (2) to limit the size of the indexers.
Decide if you need to set disk quotas for users to limit the amount of disk they can use for search results. (But be careful, because you want them to still be able to run searches and dashboards!)

0 Karma

lguinn2
Legend

Well, actually you can permanently set the number lower. But you still need to figure out what is consuming the disk, because it is probably just going to keep growing until the problem occurs again.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...