Archive
Highlighted

Why do I keep getting warning messages of 5GB even though dispatch is less than 1GB?

Contributor

Hello,

I keep getting warning messages that my dispatch directory is full (5GB) even though the dispatch dir size is less than 1 GB. And also, my queries stop running, hence I have to clean up the dispatch dir to make Splunk run again.

Kindly advise.

0 Karma
Highlighted

Re: Why do I keep getting warning messages of 5GB even though dispatch is less than 1GB?

SplunkTrust
SplunkTrust

Hi @kamal_jagga

Interesting problem

  • is is possible that a large schedule job is filling the dispatch directory but then it fails so it clears the dispatch directory before you can see its true size?
  • Is it possible that the dispatch being filled is different to the server you assume it is? Maybe post the exact error message?

Good luck with your hunt

0 Karma
Highlighted

Re: Why do I keep getting warning messages of 5GB even though dispatch is less than 1GB?

Contributor

Thanks for the reply. I had already checked for these things but wasn't of any help.

0 Karma
Highlighted

Re: Why do I keep getting warning messages of 5GB even though dispatch is less than 1GB?

SplunkTrust
SplunkTrust

The 5GB warning is not about size, it's about free space. It's possible for the dispatch directory to be completely empty and still get this warning because the disk doesn't have at least 5GB available.

The fix is to free up files elsewhere on the disk, not just the dispatch directory.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: Why do I keep getting warning messages of 5GB even though dispatch is less than 1GB?

SplunkTrust
SplunkTrust

Ah of course you are right @richgalloway ! The limit can be changed here in limits.conf if you cant make more space available dispatch_dir_warning_size = <int> https://docs.splunk.com/Documentation/Splunk/7.2.3/Admin/Limitsconf

0 Karma
Highlighted

Re: Why do I keep getting warning messages of 5GB even though dispatch is less than 1GB?

Esteemed Legend

It is not how big the volume is, it is how much free space is left. Try this command:

df /opt/splunk/var/run/splunk/dispatch/

It will show you how big the hosting volume is, how much space is used and how much is available. The Available will be > 500M if you are getting that warning/symptoms (unless you have changed the default).

0 Karma