I am receiving the following message in Splunk 6.01 "Minimum free disk space reached (5000MB) for /opt/splunk/var/run/dispatch". We also have Splunk App for ES installed. I have erased everything in the directory and after one day I receive the message again. What do I need to adjust in order to resolve this issue permanently?
I had this issue installing a heavy forwarder, I was allocated 4gb of space and after the install, I only had 3.4gb left.
After researching it, I added this to my ~/etc/system/local/server.conf:
Yes, this is low, but I'm just receiving & forwarding, and not expecting to index anything relevant here. If I run into space issues, I'll just request a few more gigs of space.
5GB is dangerously low for an enterprise system. After you optimize the TTL on your results , you may still find yourself running low on space.
See about attaching more space to your system. One of the great things about *NIX systems is that you can mount new storage directly to your var/run/dispatch directory without affecting the rest of your system.
Also if your system is using LVM volumes you can also transparently add storage without interruption.
Moreover, whichever jobs are not required you could set dispatch.ttl parameter to as low as possible.
Try suggestions from this