topic Re: Dispatch directory is full. How do we clear it up? in Splunk Search

Dispatch directory is full. How do we clear it up?

ddrillic — Mon, 04 Apr 2016 19:07:03 GMT

We get an error message in the UI, saying that the dispatch directory is full. How can we clean it? We have two SHs...

Re: Dispatch directory is full. How do we clear it up?

somesoni2 — Mon, 04 Apr 2016 19:16:33 GMT

See solutions here

https://answers.splunk.com/answers/29551/too-many-search-jobs-found-in-the-dispatch-directory.html

https://answers.splunk.com/answers/139924/where-can-i-find-documentation-for-splunkd-clean-dispatch-command.html
http://www.learnsplunk.com/splunk-kown-issues.html

Re: Dispatch directory is full. How do we clear it up?

ddrillic — Mon, 04 Apr 2016 19:23:31 GMT

Great. Can we "just" clear the files instead of moving them?

Re: Dispatch directory is full. How do we clear it up?

somesoni2 — Mon, 04 Apr 2016 20:09:22 GMT

The native splunk method only provides moving but you can create your own custom script (and can schedule it to run frequently) to remove dispatch items manually. For example, the following command will remove dispatch artifacts older than 2 hours (I run this script every 5 min). You can adjust it according to your situation.

dispatch=/var/opt/splunk/var/run/splunk/dispatch
splunkdir=/var/opt/splunk
find $dispatch -maxdepth 1 -mmin +120 2>/dev/null | while read job; do if [ ! -e "$job/save" ] ; then rm -rfv $job ; fi ; done
find $dispatch -type d -empty -name alive.token -mmin +120 2>/dev/null | xargs -i rm -Rf {}
find $splunkdir/var/run/splunk/ -type f -name "session-*" -mmin +120 2>/dev/null | xargs -i rm -Rf {}

Re: Dispatch directory is full. How do we clear it up?

ddrillic — Mon, 04 Apr 2016 20:26:06 GMT

Very very kind of you!!! it worked.

Re: Dispatch directory is full. How do we clear it up?

mmekroud — Wed, 21 Sep 2016 15:41:34 GMT

Hi somesoni2 ,
thank you for sharing this script, as I'm faced to a similar problem, i would like to know how did you implement this script,

Is a bash script? and where did you add this file ?

Thanks in advance,

regards

Re: Dispatch directory is full. How do we clear it up?

somesoni2 — Tue, 29 Sep 2020 11:06:25 GMT

This is a bash script (I named is clean_dispatch.sh). I added this script to location $Splunk_Home/bin/scripts and created a scripted input to run this every 5 mins. The reason for using scripted input was to ensure it runs on every search head, every time, as I was using SHC.

Re: Dispatch directory is full. How do we clear it up?

mmekroud — Wed, 21 Sep 2016 15:49:43 GMT

thank you for your fast reply,
i got the idea for the clean_dispatch.sh, but not for the scripted input for 5 m (i'm not so advanced in shell) so could you please show me how did you made that =?

i appreciate your help,

thank you,

regards,
Mo

Re: Dispatch directory is full. How do we clear it up?

somesoni2 — Wed, 21 Sep 2016 15:58:11 GMT

The Scripted input is Splunk feature, which is generally used to get data for custom sources. It basically can run a script at regular interval, so I'm utilizing it to run my script at frequent interval. This way I can control it's deployment better (than the OS level cron jobs).

See these for more details on Scripted input
https://docs.splunk.com/Documentation/Splunk/6.4.3/AdvancedDev/ScriptedInputsIntro
https://docs.splunk.com/Documentation/Splunk/6.4.3/AdvancedDev/ScriptSetup

Re: Dispatch directory is full. How do we clear it up?

mmekroud — Wed, 21 Sep 2016 16:05:01 GMT

Alright then, I'll check that closer,

thank you for the details,

regards,

Re: Dispatch directory is full. How do we clear it up?

AKG1_old1 — Wed, 28 Aug 2019 10:42:51 GMT

@somesoni2 : This script is really helpful. Thanks for that . Inspite of removing dispatch search older than 45 mins still count is really high (13000+) and It impacting the CPU usage. we are running some saved searched on real-time not sure if that creates multiple dispatch. One way is to increase hardware but Is there anything else we can look at ?

Re: Dispatch directory is full. How do we clear it up?

somesoni2 — Wed, 28 Aug 2019 16:03:30 GMT

Realtime searches increase the CPU utilization a lot as they hold on to one core forever. A real-time scheduled search doesn't stop, so it'll hold on to those system resources. I would suggest if possible, convert them to historical saved search and run them at a proper frequency. See this post for more explanation

https://answers.splunk.com/answers/100823/how-to-set-up-real-time-search-as-saved-search.html
https://docs.splunk.com/Documentation/Splunk/7.3.1/Search/Aboutrealtimesearches

Re: Dispatch directory is full. How do we clear it up?

AKG1_old1 — Wed, 28 Aug 2019 16:22:53 GMT

@somesoni : Thank you for response. we have converted some real-time searches to scheduled searches. but real issue is we are running many scheduled search in parallel. We enhance your script to run every minute and removing older than 1 minutes of dispatch dir. Now getting better cpu performance. Not sure if its advisible or not but not seeing any negative impact.

Re: Dispatch directory is full. How do we clear it up?

somesoni2 — Wed, 28 Aug 2019 16:27:12 GMT

You probably need to scale up your search heads, so that your concurrency limits can be increased. You'll see problem if someone tries to see the result of a saved search (using loadjob or from job manager) as the artifacts are deleted, those search results won't be there. Again, it depends if someone uses it that way.

Re: Dispatch directory is full. How do we clear it up?

AKG1_old1 — Wed, 28 Aug 2019 16:35:45 GMT

@somesoni2 : Agreed !! we are planning to increase hardware. I thought so it will impact load job but we don't use loadjob anywhere so this is working for us. Thank you so much for clarification and help 🙂

Re: Dispatch directory is full. How do we clear it up?

joshiro — Fri, 03 Feb 2023 16:24:40 GMT

We need to clean the dispatch directory in a SH clustered environment.

We didnt found any best practices for the clean-dispatch command and the Splunk documentation doesnt help either.
https://docs.splunk.com/Documentation/Splunk/9.0.3/Search/Dispatchdirectoryandsearchartifacts

Should we run the clean-dispatch command node per node? Stop node, clean-dispatch, start node?
Or should we stop the whole SH cluster, then clean-dispatch each node, and then start the nodes?