Splunk Search

Dispatch directory is full. How do we clear it up?

ddrillic
Ultra Champion

We get an error message in the UI, saying that the dispatch directory is full. How can we clean it? We have two SHs...

Tags (1)
1 Solution

ddrillic
Ultra Champion

Great. Can we "just" clear the files instead of moving them?

0 Karma

somesoni2
Revered Legend

The native splunk method only provides moving but you can create your own custom script (and can schedule it to run frequently) to remove dispatch items manually. For example, the following command will remove dispatch artifacts older than 2 hours (I run this script every 5 min). You can adjust it according to your situation.

dispatch=/var/opt/splunk/var/run/splunk/dispatch
splunkdir=/var/opt/splunk
find $dispatch -maxdepth 1 -mmin +120 2>/dev/null | while read job; do if [ ! -e "$job/save" ] ; then rm -rfv $job ; fi ; done
find $dispatch -type d -empty -name alive.token -mmin +120 2>/dev/null | xargs -i rm -Rf {}
find $splunkdir/var/run/splunk/ -type f -name "session-*" -mmin +120 2>/dev/null | xargs -i rm -Rf {}

mmekroud
Explorer

Hi somesoni2 ,
thank you for sharing this script, as I'm faced to a similar problem, i would like to know how did you implement this script,

Is a bash script? and where did you add this file ?

Thanks in advance,

regards

0 Karma

somesoni2
Revered Legend

This is a bash script (I named is clean_dispatch.sh). I added this script to location $Splunk_Home/bin/scripts and created a scripted input to run this every 5 mins. The reason for using scripted input was to ensure it runs on every search head, every time, as I was using SHC.

mmekroud
Explorer

thank you for your fast reply,
i got the idea for the clean_dispatch.sh, but not for the scripted input for 5 m (i'm not so advanced in shell) so could you please show me how did you made that =?

i appreciate your help,

thank you,

regards,
Mo

somesoni2
Revered Legend

The Scripted input is Splunk feature, which is generally used to get data for custom sources. It basically can run a script at regular interval, so I'm utilizing it to run my script at frequent interval. This way I can control it's deployment better (than the OS level cron jobs).

See these for more details on Scripted input
https://docs.splunk.com/Documentation/Splunk/6.4.3/AdvancedDev/ScriptedInputsIntro
https://docs.splunk.com/Documentation/Splunk/6.4.3/AdvancedDev/ScriptSetup

mmekroud
Explorer

Alright then, I'll check that closer,

thank you for the details,

regards,

0 Karma

agoyal
Builder

@somesoni2 : This script is really helpful. Thanks for that . Inspite of removing dispatch search older than 45 mins still count is really high (13000+) and It impacting the CPU usage. we are running some saved searched on real-time not sure if that creates multiple dispatch. One way is to increase hardware but Is there anything else we can look at ?

0 Karma

somesoni2
Revered Legend

Realtime searches increase the CPU utilization a lot as they hold on to one core forever. A real-time scheduled search doesn't stop, so it'll hold on to those system resources. I would suggest if possible, convert them to historical saved search and run them at a proper frequency. See this post for more explanation

https://answers.splunk.com/answers/100823/how-to-set-up-real-time-search-as-saved-search.html
https://docs.splunk.com/Documentation/Splunk/7.3.1/Search/Aboutrealtimesearches

agoyal
Builder

@somesoni : Thank you for response. we have converted some real-time searches to scheduled searches. but real issue is we are running many scheduled search in parallel. We enhance your script to run every minute and removing older than 1 minutes of dispatch dir. Now getting better cpu performance. Not sure if its advisible or not but not seeing any negative impact.

0 Karma

somesoni2
Revered Legend

You probably need to scale up your search heads, so that your concurrency limits can be increased. You'll see problem if someone tries to see the result of a saved search (using loadjob or from job manager) as the artifacts are deleted, those search results won't be there. Again, it depends if someone uses it that way.

agoyal
Builder

@somesoni2 : Agreed !! we are planning to increase hardware. I thought so it will impact load job but we don't use loadjob anywhere so this is working for us. Thank you so much for clarification and help 🙂

0 Karma

ddrillic
Ultra Champion

Very very kind of you!!! it worked.

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...