Splunk Enterprise Security

Unable to run Security Posture in the Splunk Cloud sandbox with error "The minimum free disk space (2000MB) reached"?

New Member
Search not executed: The minimum free disk space (2000MB) reached for /opt/splunk/var/run/splunk/dispatch. user=wtaddis. 

Splunk Version
6.3.1511

Splunk Build
8effae892620

0 Karma

SplunkTrust
SplunkTrust

This could mean that your dispatch folder is "full" which will prevent you from doing any searches. This became full because too many searches we're going on in parallel and you don't have enough room on the file system. You can manually clear these files without any harm, this will just kill the search

Or this means that your opt drive is full. Most likely your coldb is retaining a lot of old files and not moving them to the frozen bucket. Go to /opt/splunk/var/lib/splunk/_internaldb and do a du -sh * and see what is taking up space

You could also go into the config file and decrease the file size needed which will temporarily fix your problem, but you will have the same issue again very quickly. This is in server.conf under the [diskUsage] stanza.. It should be like minFreeSpace =xx

Go look in your db and see what files are taking lots of room and delete some. You should then go to your settings/indexes and set a max size for your cold bucket to prevent this in the future.

0 Karma

SplunkTrust
SplunkTrust

I also want to describe what the dispatch folder does for more clarity..

The dispatch dir will house "artifacts" and these searches will be "cached" in the dispatch directory so you can load up searches faster. I believe the scheduled searches are relative to the timespan of the search, so if you have a long timespan then this will live in the dispatch folder for a longer period of time (Could be days). So to sum it up, if you have a lot of scheduled searches AND they have a big timespan specified, then this will quickly clog up your dispatch folder. So you will need to increase the size, decrease the amount of scheduled searches, decrease the timespan in those scheduled searches or decrease the minimum free disk space

Community Manager
Community Manager

Here's an answer from a previous post on this topic for further reading 🙂
https://answers.splunk.com/answers/213571/what-causes-too-many-search-jobs-found-in-the-disp.html#an...

SplunkTrust
SplunkTrust

Ahh 2p, that's right!

0 Karma

New Member

Thanks. Since this is a Splunk Enterprise Security Workshop located in the Splunk Cloud would the configuration take place in Splunk's infrastrucure.? Thanks again.

0 Karma

SplunkTrust
SplunkTrust

Yes on the indexer

0 Karma

New Member

This is for a Splunk Enterprise Security Workshop

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!