I had this error below:
search:262 - reason="Search not executed: Dispatch Manager: The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch. user=admin.", concurrencycategory="historical", concurrencycontext="userinstance-wide", currentconcurrency=0, concurrency_limit=5000
I do not want to change the limit 5000MB.
I already had deleted some logs in /opt/splunk/var/log/splunk like metrics.log.* but it did not resolve the issue.
Also try the clean-dispatch command, and also deleted the files inside the /opt/splunk/var/run/splunk/dispatch.
Is there any suggestion you could share with me excluding the change the set limit of 5000MB?
Best answer: redo your search head so that
/opt/splunk/var is in its own disk volume so that nothing else can take up this space. Some people even go so var as to put
/opt/splunk/var/run/dispatch into its own volume.
The next best answer: add more disk space to to the volume that contains the
OK answer: Reduce the TTL for search artifacts so things auto-purge more quickly as described here: https://www.splunk.com/blog/2012/09/12/how-long-does-my-search-live-default-search-ttl.html
The WORST answer: reduce this quota in
Server settings ->
General settings ->
Pause indexing if free disk space (in MB) falls below.