Why do I keep getting warning messages of 5GB even though dispatch is less than 1GB?

[kamal_jagga]

Hello,

I keep getting warning messages that my dispatch directory is full (5GB) even though the dispatch dir size is less than 1 GB. And also, my queries stop running, hence I have to clean up the dispatch dir to make Splunk run again.

Kindly advise.

[woodcock]

It is not how big the volume is, it is how much free space is left. Try this command:

df /opt/splunk/var/run/splunk/dispatch/

It will show you how big the hosting volume is, how much space is used and how much is available. The Available will be > 500M if you are getting that warning/symptoms (unless you have changed the default).

[chrisyounger]

Ah of course you are right @richgalloway ! The limit can be changed here in limits.conf if you cant make more space available dispatch_dir_warning_size = <int> https://docs.splunk.com/Documentation/Splunk/7.2.3/Admin/Limitsconf

[richgalloway]

The 5GB warning is not about size, it's about free space. It's possible for the dispatch directory to be completely empty and still get this warning because the disk doesn't have at least 5GB available.

The fix is to free up files elsewhere on the disk, not just the dispatch directory.

[chrisyounger]

Hi @kamal_jagga

Interesting problem

• is is possible that a large schedule job is filling the dispatch directory but then it fails so it clears the dispatch directory before you can see its true size?
• Is it possible that the dispatch being filled is different to the server you assume it is? Maybe post the exact error message?

Good luck with your hunt

[kamal_jagga]

Thanks for the reply. I had already checked for these things but wasn't of any help.