Unable to run Security Posture in the Splunk Cloud sandbox with error "The minimum free disk space (2000MB) reached"?

[wtaddis]

Search not executed: The minimum free disk space (2000MB) reached for /opt/splunk/var/run/splunk/dispatch. user=wtaddis. 

Splunk Version
6.3.1511

Splunk Build
8effae892620

[skoelpin]

This could mean that your dispatch folder is "full" which will prevent you from doing any searches. This became full because too many searches we're going on in parallel and you don't have enough room on the file system. You can manually clear these files without any harm, this will just kill the search

Or this means that your opt drive is full. Most likely your coldb is retaining a lot of old files and not moving them to the frozen bucket. Go to /opt/splunk/var/lib/splunk/_internaldb and do a du -sh * and see what is taking up space

You could also go into the config file and decrease the file size needed which will temporarily fix your problem, but you will have the same issue again very quickly. This is in server.conf under the [diskUsage] stanza.. It should be like minFreeSpace =xx

Go look in your db and see what files are taking lots of room and delete some. You should then go to your settings/indexes and set a max size for your cold bucket to prevent this in the future.

[skoelpin]

I also want to describe what the dispatch folder does for more clarity..

The dispatch dir will house "artifacts" and these searches will be "cached" in the dispatch directory so you can load up searches faster. I believe the scheduled searches are relative to the timespan of the search, so if you have a long timespan then this will live in the dispatch folder for a longer period of time (Could be days). So to sum it up, if you have a lot of scheduled searches AND they have a big timespan specified, then this will quickly clog up your dispatch folder. So you will need to increase the size, decrease the amount of scheduled searches, decrease the timespan in those scheduled searches or decrease the minimum free disk space

[ppablo]

Here's an answer from a previous post on this topic for further reading 🙂
https://answers.splunk.com/answers/213571/what-causes-too-many-search-jobs-found-in-the-disp.html#an...

[skoelpin]

Ahh 2p, that's right!

[wtaddis]

Thanks. Since this is a Splunk Enterprise Security Workshop located in the Splunk Cloud would the configuration take place in Splunk's infrastrucure.? Thanks again.

[skoelpin]

Yes on the indexer

[wtaddis]

This is for a Splunk Enterprise Security Workshop