Can no longer run searches, error messages occurring

[jmartelon]

I am unable to search,

I had a few error messages come up:

Dispatch command: the minimum free disk space 8000MB reached for /opt/splunk/var/run/splunk/dispatch

Failed to start KV Store process. See mongod.og and splunkd.log for details.

Disk Monitor: The Index processor has paused data flow. Current free disk space on partition '/' has fallen to 4485MB, below the minimum of 8000MB. Data writes to index path '/opt/splunk/var/lib/splunk/audit/db' cannot safely proceed. Increase disk space on partition '/'.

So Currently on /, I am only at 59% usage, and I am not sure why I am seeing this log or error message..

Please assist!

[493669]

Try this in serer.conf of $splunk_Home/etc/system/local/:

 [diskUsage]
 minFreeSpace = 500

Limits for controlling disk space in Splunk can be changed

The relevant stanza and parameter of interest in server.conf is:

[diskUsage]
minFreeSpace =
For more details please look here:

http://docs.splunk.com/Documentation/Splunk/6.2.1/Indexer/Setlimitsondiskusage

This can be changed on any Splunk installations as explained on the online documentation: "for all installations, including forwarders, you must have a minimum of 5GB of hard disk space available in addition to the space required for any indexes." The default is 5000 and this value can be changed as explained before.

For more details, please check here:

http://docs.splunk.com/Documentation/Splunk/6.2.1/Installation/Systemrequirements#Recommended_hardwa...

Hope this helps.

[skoelpin]

You have two issues here. The first one is the dispatch directory queueing which is pausing your searches.. This could mean you have too much search activity or not enough hardware. You can clear out the files from the dispatch directory or wait for them to clear on their own as the TTL is relative to the length of the search.

Second issue is you have less than the minimum amount of disk space available that is configured in Splunk. This is a good thing to have because it stops Splunk before reaching 100% full. You should look at the root directory and see how much space is available. Splunk wants atleast 5GB and your claiming to be at 59% usage. If you have a small enough drive then this can absolutely be true. Perhaps the cached searches in your dispatch directory caused the increase in disk space

https://www.splunk.com/blog/2012/09/10/a-quick-tour-of-a-dispatch-directory.html

[jmartelon]

Thank you for the information, unfortunately, I am still not able to search. I did clear the dispatch directory, and restarted splunk and still cannot search. I am still getting the error messages, and I have checked root and I am completely fine on space.

If you have any more ideas, please let me know.

[skoelpin]

What error are you getting?

You should look at your internal logs for errors

index=_internal sourcetype=splunkd

[jmartelon]

I can't run the above search.

I am still getting the same error messages... The minimum free disk space 8000mb reached for /opt/splunk/var/run/splunk/dispatch

And

The index processor has paused data flow. Current free disk space on partition / has fallen to 4603Mb below the minimum of 5000MB Data writes to index path /opt/splunk/var/lib/splunk/audit/db' cannot safely proceed.

I'm also seeing failed to start KV Store process. See mongod.log and splunkd.log for details.

KV store changed status to failed. KVSTore process terminated.

[maciep]

maybe this is relevant? i don't think it's percentage based, but free space based.

http://docs.splunk.com/Documentation/Splunk/7.0.1/Indexer/Setlimitsondiskusage