Splunk Enterprise Security

Unable to run Security Posture in the Splunk Cloud sandbox with error "The minimum free disk space (2000MB) reached"?

wtaddis
New Member
Search not executed: The minimum free disk space (2000MB) reached for /opt/splunk/var/run/splunk/dispatch. user=wtaddis. 

Splunk Version
6.3.1511

Splunk Build
8effae892620

0 Karma

skoelpin
SplunkTrust
SplunkTrust

This could mean that your dispatch folder is "full" which will prevent you from doing any searches. This became full because too many searches we're going on in parallel and you don't have enough room on the file system. You can manually clear these files without any harm, this will just kill the search

Or this means that your opt drive is full. Most likely your coldb is retaining a lot of old files and not moving them to the frozen bucket. Go to /opt/splunk/var/lib/splunk/_internaldb and do a du -sh * and see what is taking up space

You could also go into the config file and decrease the file size needed which will temporarily fix your problem, but you will have the same issue again very quickly. This is in server.conf under the [diskUsage] stanza.. It should be like minFreeSpace =xx

Go look in your db and see what files are taking lots of room and delete some. You should then go to your settings/indexes and set a max size for your cold bucket to prevent this in the future.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

I also want to describe what the dispatch folder does for more clarity..

The dispatch dir will house "artifacts" and these searches will be "cached" in the dispatch directory so you can load up searches faster. I believe the scheduled searches are relative to the timespan of the search, so if you have a long timespan then this will live in the dispatch folder for a longer period of time (Could be days). So to sum it up, if you have a lot of scheduled searches AND they have a big timespan specified, then this will quickly clog up your dispatch folder. So you will need to increase the size, decrease the amount of scheduled searches, decrease the timespan in those scheduled searches or decrease the minimum free disk space

ppablo
Retired

Here's an answer from a previous post on this topic for further reading 🙂
https://answers.splunk.com/answers/213571/what-causes-too-many-search-jobs-found-in-the-disp.html#an...

skoelpin
SplunkTrust
SplunkTrust

Ahh 2p, that's right!

0 Karma

wtaddis
New Member

Thanks. Since this is a Splunk Enterprise Security Workshop located in the Splunk Cloud would the configuration take place in Splunk's infrastrucure.? Thanks again.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Yes on the indexer

0 Karma

wtaddis
New Member

This is for a Splunk Enterprise Security Workshop

0 Karma
Get Updates on the Splunk Community!

Thanks for the Memories! Splunk University, .conf24, and Community Connections

Thank you to everyone in the Splunk Community who joined us for .conf24 – starting with Splunk University and ...

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...