Dispatch directory is full. How do we clear it up?

[ddrillic]

We get an error message in the UI, saying that the dispatch directory is full. How can we clean it? We have two SHs...

[somesoni2 · Accepted solution]

See solutions here

https://answers.splunk.com/answers/29551/too-many-search-jobs-found-in-the-dispatch-directory.html

https://answers.splunk.com/answers/139924/where-can-i-find-documentation-for-splunkd-clean-dispatch-...
http://www.learnsplunk.com/splunk-kown-issues.html

[joshiro]

We need to clean the dispatch directory in a SH clustered environment.

We didnt found any best practices for the clean-dispatch command and the Splunk documentation doesnt help either.
https://docs.splunk.com/Documentation/Splunk/9.0.3/Search/Dispatchdirectoryandsearchartifacts

Should we run the clean-dispatch command node per node? Stop node, clean-dispatch, start node?
Or should we stop the whole SH cluster, then clean-dispatch each node, and then start the nodes?

[ddrillic]

Great. Can we "just" clear the files instead of moving them?

[somesoni2]

The native splunk method only provides moving but you can create your own custom script (and can schedule it to run frequently) to remove dispatch items manually. For example, the following command will remove dispatch artifacts older than 2 hours (I run this script every 5 min). You can adjust it according to your situation.

dispatch=/var/opt/splunk/var/run/splunk/dispatch
splunkdir=/var/opt/splunk
find $dispatch -maxdepth 1 -mmin +120 2>/dev/null | while read job; do if [ ! -e "$job/save" ] ; then rm -rfv $job ; fi ; done
find $dispatch -type d -empty -name alive.token -mmin +120 2>/dev/null | xargs -i rm -Rf {}
find $splunkdir/var/run/splunk/ -type f -name "session-*" -mmin +120 2>/dev/null | xargs -i rm -Rf {}

[mmekroud]

Hi somesoni2 ,
thank you for sharing this script, as I'm faced to a similar problem, i would like to know how did you implement this script,

Is a bash script? and where did you add this file ?

Thanks in advance,

regards

[somesoni2]

This is a bash script (I named is clean_dispatch.sh). I added this script to location $Splunk_Home/bin/scripts and created a scripted input to run this every 5 mins. The reason for using scripted input was to ensure it runs on every search head, every time, as I was using SHC.

[mmekroud]

thank you for your fast reply,
i got the idea for the clean_dispatch.sh, but not for the scripted input for 5 m (i'm not so advanced in shell) so could you please show me how did you made that =?

i appreciate your help,

thank you,

regards,
Mo

[somesoni2]

The Scripted input is Splunk feature, which is generally used to get data for custom sources. It basically can run a script at regular interval, so I'm utilizing it to run my script at frequent interval. This way I can control it's deployment better (than the OS level cron jobs).

See these for more details on Scripted input
https://docs.splunk.com/Documentation/Splunk/6.4.3/AdvancedDev/ScriptedInputsIntro
https://docs.splunk.com/Documentation/Splunk/6.4.3/AdvancedDev/ScriptSetup

[mmekroud]

Alright then, I'll check that closer,

thank you for the details,

regards,

[AKG1_old1]

@somesoni2 : This script is really helpful. Thanks for that . Inspite of removing dispatch search older than 45 mins still count is really high (13000+) and It impacting the CPU usage. we are running some saved searched on real-time not sure if that creates multiple dispatch. One way is to increase hardware but Is there anything else we can look at ?

[somesoni2]

Realtime searches increase the CPU utilization a lot as they hold on to one core forever. A real-time scheduled search doesn't stop, so it'll hold on to those system resources. I would suggest if possible, convert them to historical saved search and run them at a proper frequency. See this post for more explanation

https://answers.splunk.com/answers/100823/how-to-set-up-real-time-search-as-saved-search.html
https://docs.splunk.com/Documentation/Splunk/7.3.1/Search/Aboutrealtimesearches

[AKG1_old1]

@somesoni : Thank you for response. we have converted some real-time searches to scheduled searches. but real issue is we are running many scheduled search in parallel. We enhance your script to run every minute and removing older than 1 minutes of dispatch dir. Now getting better cpu performance. Not sure if its advisible or not but not seeing any negative impact.

[somesoni2]

You probably need to scale up your search heads, so that your concurrency limits can be increased. You'll see problem if someone tries to see the result of a saved search (using loadjob or from job manager) as the artifacts are deleted, those search results won't be there. Again, it depends if someone uses it that way.

[AKG1_old1]

@somesoni2 : Agreed !! we are planning to increase hardware. I thought so it will impact load job but we don't use loadjob anywhere so this is working for us. Thank you so much for clarification and help 🙂

[ddrillic]

Very very kind of you!!! it worked.