topic Splunk Search Head : "Search not executed: The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch" in Splunk Search

Splunk Search Head : "Search not executed: The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch"

ramaswamy — Tue, 19 Apr 2016 14:55:02 GMT

From Splunk Web, when I run a search, I receive the following message

Search not executed: The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch

In Settings->ServerSettings->GeneralSettings, I have the Splunk Search Head host configured.

Splunk Search Head is having a total disk capacity of 30GB and currently 25GB is used up.

My setup has
1) One master node
2) One Search head
3) Three Peer nodes

Does the above error mean that, I am trying to index the logs of Search Head host and hence I am running short of disk space?

/volr/splunk/defaultdb/db/ seems to use 12GB of space and I believe that it is the index data.

Re: Splunk Search Head : "Search not executed: The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch"

richgalloway — Tue, 19 Apr 2016 17:16:29 GMT

Your disk has only 5GB free (or less) and Splunk requires at least 5GB free (5000 MB). That is why searches are not executed. To resume searching, you must release some disk space. Look for old jobs in the dispatch directory that haven't been cleaned up.

Re: Splunk Search Head : "Search not executed: The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch"

ramaswamy — Tue, 19 Apr 2016 17:47:37 GMT

Does Splunk SearchHead index data? In Webui, I see a message that indexing is stopped. Also, in Settings - ServerSettings - GeneralSettings, I see Index option.

Re: Splunk Search Head : "Search not executed: The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch"

richgalloway — Tue, 19 Apr 2016 18:13:45 GMT

Search heads can index data and yours appears to do so. That's fine for summary indexes, but raw data should be on your indexer(s). Double-check your input settings and your forwarders to make sure data is going to the right place.

Re: Splunk Search Head : "Search not executed: The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch"

vasanthmss — Tue, 19 Apr 2016 19:04:38 GMT

richgalloway is correct. go to your server and check the disk mounted for Splunk. you can use
df -ah to validate in the unix/linux environment.

Check the below answer to clean up the dispatch,

https://answers.splunk.com/answers/139924/where-can-i-find-documentation-for-splunkd-clean-dispatch-command.html

Try to use SOS / Splunk Health Overview App from Splunk base. to know further about your environment.