topic Dispatch directory full in Deployment Architecture

Dispatch directory full

enzomialich — Fri, 01 Nov 2024 21:25:45 GMT

Hi, if an environment encounters the error 'Search not executed: The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch' multiple times (meaning the issue persists even after cleaning the dispatch directory), what corrective actions should be taken? Should the dispatch directory be cleaned regularly?
This is for a standalone environment.

Re: Dispatch directory full

burwell — Fri, 01 Nov 2024 21:54:03 GMT

Here's some good information about the dispatch directory: https://docs.splunk.com/Documentation/Splunk/9.3.1/Search/Dispatchdirectoryandsearchartifacts

Splunk normally does age things out but read the doc above.

Perhaps the disk is full for other reasons? https://community.splunk.com/t5/Splunk-Search/Splunk-says-dispatch-directory-is-full-but-when-I-go-to-the/m-p/370243

One thing that can cause your dispatch directory to grow is if you adjust the time to live (TTL) of jobs.

Re: Dispatch directory full

PickleRick — Fri, 01 Nov 2024 21:56:48 GMT

It's up to your OS and/or Splunk admins to solve. For some reason the filesystem on which the dispatch directory is located is filled up to the brim. It might be just the dispatch data but if it's on the same filesystem as - for example - Splunk's internal logs and maybe OS logs and other stuff there could be other places you need to be looking for free space.

Re: Dispatch directory full

enzomialich — Fri, 01 Nov 2024 22:54:48 GMT

edited

Re: Dispatch directory full

enzomialich — Fri, 01 Nov 2024 22:54:21 GMT

If I put a lower value on TTL for the dispatch directory - this would be a good idea in this case?