Received error this morning on one of our non-distributed search head:
The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch.
Nothing works, cannot search, dashboards are non-functional.
Searching produces this error:
Search not executed: The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch. user=admin., concurrency_category="historical", concurrency_context="user_instance-wide", current_concurrency=0, concurrency_limit=5000
I did quite a bit of digging in the community and found the following on my instances, non-distributed:
Dispatch
Tried the clean-dispatch command on our bloated 8873 count in /opt/splunk/var/run/splunk/dispatch
Shut down splunk even run in sudo, results in error of Permission denied
Ran command: ./splunk cmd splunkd clean-dispatch /temp -1day
bundle files
distsearches.conf has no maxbundlesize addressing the large .bundle files in /opt/splunk/var/run
If I delete out the bundle files above, I can search for alittle bit on the search head, but then it craps out.
Now, I am at a loss after reading so many articles, how-tos and docs. I'm not a splunk guy, but I am trying to get this stable.
