The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch on an Indexer

[aoliullah]

Hi. I get the following error on one of my indexers.

The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch.

Having looked at the folder, all the files are from today. Is it safe for me to clean this folder? I understand that you could modify parameters in limits.conf, however, I want the indexer's minimum disk space to be left at 5000MB which I believe is safe?

Could you please advise?

[woodcock]

An alternate approach is to add more disk space to the dispatch directory. You can do this like this:

1: stop splunk on the search head
2: mount a new disk with a TON of space to anywhere but let's say to /mnt/MegaDiskForSplunkDispatch (ensure automount with fstab, etc.).
3: make new mount have same ownership ( chown ) and same permission ( chmod ) as indicated by:

ls -al $SPLUNK_HOME/var/run/splunk | grep dispatch

4: move all dispatch files:

mv $SPLUNK_HOME/var/run/splunk/dispatch/* /mnt/MegaDiskForSplunkDispatch/

5: remove the dispatch directory

rmdir $SPLUNK_HOME/var/run/splunk/dispatch

6: replace with softlink:

ln -fs /mnt/MegaDiskForSplunkDispatch/ $SPLUNK_HOME/var/run/splunk/dispatch/

7: Fix the permissions

chmod 711 $SPLUNK_HOME/var/run/splunk/dispatch/

8: restart splunk

[koshyk]

My advice is to increase the filesystem space where /opt/splunk is residing
the clean-dispatch is a temporary fix only

In my experience for (minimum)
- Standalone system requires 20GB for /opt/splunk
- Cluster system requires 50GB for /opt/splunk (for bundle replication)

[MuS]

Hi aoliullah,

Have a look at the docs http://docs.splunk.com/Documentation/Splunk/6.5.3/Search/Dispatchdirectoryandsearchartifacts#Clean_u... and learn about the clean-dispatch command. Otherwise if you no longer need the search artifacts you can actually just delete the files.

Hope this helps ...

cheers, MuS

[aoliullah]

Hi MuS - I have familiarise myself with the commands already. I wanted to find out how I could determine if I need those files or not. May be they are being used by a current search running in the background? Could you advise?

[MuS]

No, because this all depends on your searches/use case.
Just my 2cents, I usually remove anything that is older than 15 minutes and never had any problems/troubles. Start deleting/removing the oldest ones only and slowly decrease the age.

[aoliullah]

But all the files seem to be around 5 to 10 mins old? Hence why I'm concerned if deleting it would cause any issues. Btw I'm talking about the dispatch directory on the indexer. Not sure why it should generate files in the first place as it's not used to carry out searches.

Plus they just seem to update in time constantly.

[brreeves_splunk]

If a search builds out lookup tables, these files can be VERY large. If they're created in the last 5-10 minutes, then someone is running searches right now with LARGE results/outputs.

Did you recently install any new apps?

[aoliullah]

I have just taken over a project and have no clue about what has been done in the past. The lead consultant has left last night so Just trying to fix a list of errors and this one looked like it needed immediate attention.

[brreeves_splunk]

If it continues to rebuild those large files, you'll want to look at the SID (search ID) to find out which one is creating these large outputs. Either that, or raise the 5000MB limit on the dispatch folder with the dispatch_dir_warning_size = <int> setting.

http://docs.splunk.com/Documentation/Splunk/6.5.3/Admin/Limitsconf#.5Bsearch.5D

[aoliullah]

Right, but the actual size of the dispatch directory is only 1230 MB so not really large. Plus the fact it's on the indexer (which no one should be using to run any searches), isn't it wiser to go ahead and just clean it?

[brreeves_splunk]

It is OK to go ahead and clean it, but that's just a bandaid on a potentially tedious process.

This conversation is going into deployment planning (which I understand since you just took it over). Whether you have a support contract or not, I might suggest contacting your Account Manager and seeing if you can talk to an SE or if you have any training credits on your account to get a deeper understanding of the intricacies of Splunk.

That dispatch folder is replicated across Clusters and, as I said earlier, is used as cache so it is going to continue to grow and shrink based on search activity.

The next question would be, where is the rest of the space in your drive going? Is this just the OS drive? Do we need to clean up other things to free up OS space? Is it safe to raise that 5000MB limit (which it would be if your drive is only 128GB to begin with)?

I hope all of this helps. 🙂

[aoliullah]

Thank you. I'll try to address it all. But to delete it does the clean-dispatch work? Looks like it just moves it to a different folder?

[brreeves_splunk]

It does...just to make 100% sure you didn't need any of it. That's why I believe the documentation suggests moving it to /tmp

Once you confirm all is well in Splunk, you can clear /tmp

[aoliullah]

Thank you! 🙂

[brreeves_splunk]

These files are generated when running searches or saved searches. They can safely be deleted, and will be regenerated when you re-run those searches again. Think of it as web-cache for Splunk Searches. 🙂