topic Re: Too many search jobs found in the dispatch directory in Reporting

Too many search jobs found in the dispatch directory

vbumgarner — Wed, 24 Aug 2011 15:54:13 GMT

i'm seeing this message after firing up backfill data in Splunk Deployment Monitor:

Too many search jobs found in the dispatch directory (found=3692, warning level=2000). This could negatively impact Splunk's performance, consider removing some of the old search jobs.

Is this dangerous? Can I manually clear out var/run/splunk/dispatch when it's done?

Re: Too many search jobs found in the dispatch directory

jbsplunk — Wed, 24 Aug 2011 22:15:24 GMT

If you see this error you can manually clear out any jobs in the dispatch folder. I would probably recommend you start with the older ones. The only downside is that the artifacts of those saved searches which populated the summary index won't be around once you clear the dispatch directory. Since the data you are after is in the summary index, it doesn't matter. Any artifacts you eliminate will be regenerated at the next scheduled run time for a particular saved search.

Re: Too many search jobs found in the dispatch directory

the_wolverine — Thu, 20 Oct 2011 02:38:15 GMT

Conversely, is there some way to raise the threshold if you actually running that many searches?

Re: Too many search jobs found in the dispatch directory

brianokelly — Wed, 09 Nov 2011 22:34:50 GMT

To be honest, I would expect splunk to clean these out. It seems like it should be an automatic house keeping task. We are starting to see this quite regularly in our deployment.

Re: Too many search jobs found in the dispatch directory

Ellen — Fri, 02 Dec 2011 20:07:42 GMT

To add to jbsplunk's answer:

The number of directories relating to the search artifacts in the Dispatch directory can potentially affect search performance since we have to scan each of the directories to determine if the artifacts are present or not.

The UI warning message about Dispatch directories being > 2K is new to 4.2.3. There isn’t any hard limit of 2K that is impacting anything as the warning is what was implemented as a best practice to start to review search jobs in general eg. is there a scheduled search that has an excessively long ttl?

You can change that 2K to a higher number so the warning takes longer to display via the

[search] stanza in your
$SPLUNK_HOME/etc/system/local/limits.conf

From the $SPLUNK_HOME/etc/system/README/limits.conf.spec:

[search]
dispatch_dir_warning_size = <int>

* The number of jobs in the dispatch directory when to issue a bulletin message warning that performance could be impacted
* Defaults to 2000

The appropriate number of Dispatch directories that should be set before the performance is impacted would vary per environment as it would depend on variables such as the volume and type of searches being run, what are the ttl etc.

If you find subdirectories in $SPLUNK_HOME/var/run/splunk/dispatch beyond 24hrs of their last modtime AND the subdirectory does not contain BOTH info.csv and status.csv files, that is considered a failed search job and that subdirectory can be safely removed. We expect this should be automatically performed by the dispatch reaper starting in 4.3

In the meanwhile outside of your own cron/scripting, there is an option where you can move subdirectories based on a timeline you have determined is acceptable out of the Dispatch directory .

Below is the usage information.
Use this command to move jobs whose last modification time is earlier than the specified time from the dispatch directory to the specified destination directory.

usage: $SPLUNK_HOME/bin/splunk cmd splunkd clean-dispatch {destination directory where to move jobs} {latest job mod time}

The destination directory must be on the same partition/filesystem as the dispatch directory.

example: splunk cmd splunkd clean-dispatch /opt/splunk/old-dispatch-jobs/ -1month
example: splunk cmd splunkd clean-dispatch /opt/splunk/old-dispatch-jobs/ -10d@d
example: splunk cmd splunkd clean-dispatch /opt/splunk/old-dispatch-jobs/ 2011-06-01T12:34:56.000-07:00

There are future enhancements to manage search job cleanup.

Re: Too many search jobs found in the dispatch directory

the_wolverine — Fri, 02 Dec 2011 23:19:41 GMT

Our issue looks like a bug in jobs. Certain jobs were not showing as completed and therefore hung around on disk until they expired. Issue was reported to Splunk Support and no word as to whether they will mark this as a bug or not. For now we have deleted all these old scheduled searches (probably some configuration issue after upgrade) and recreated them by hand.

Re: Too many search jobs found in the dispatch directory

responsys_cm — Mon, 28 Sep 2020 11:57:41 GMT

I had the same problem. I ran the clean-dispatch command and for a few results got the following:

Could not move /opt/splunk/var/run/splunk/dispatch/scheduler_cmerchantsearch_SW50ZXJhY3QgU3VzcGljaW91cyBTb3VyY2UgSVAgQWxlcnQ_at_1339130700_2aeaae3b6e48c487 to /space/splunktmp/schedulercmerchant_search_SW50ZXJhY3QgU3VzcGljaW91cyBTb3VyY2UgSVAgQWxlcnQ_at_1339130700_2aeaae3b6e48c487. Invalid cross-device link

Re: Too many search jobs found in the dispatch directory

jrodman — Thu, 24 Jan 2013 16:16:53 GMT

I updated the answer; for the current limitations of clean-dispatch, destdir must be on the same file system as the dispatch dir.

Re: Too many search jobs found in the dispatch directory

hugo_vazquez — Thu, 13 Jun 2013 22:46:55 GMT

I ran clean-dispatch command and it moves them to old-dispatch directory, do I really need to keep them or can I just delete them?

Re: Too many search jobs found in the dispatch directory

ACNmateen — Tue, 17 Sep 2013 08:59:15 GMT

are there any latest updates as to how to clean up those extra jobs running and giving errors ?

Re: Too many search jobs found in the dispatch directory

wsnyder2 — Tue, 21 Apr 2015 19:27:22 GMT

great question ! We have had Splunk for 2 years, I have never had anyone ask me for search results from the old-dispatch-jobs directory. So in my mind that means delete them say after a week. : - )

Re: Too many search jobs found in the dispatch directory

srisahitya_v — Thu, 14 May 2015 05:43:05 GMT

to add answer to the ellen,

some times, it won't delete because of <old_dispatch> directory what we created to move dispatch jobs may full. so create one more and move them. it works.

Re: Too many search jobs found in the dispatch directory

srisahitya_v — Mon, 08 Jun 2015 07:56:53 GMT

might be "real time alerts" also a cause to produce many dispatch files.

for my case, one real time alert was triggering 3-5 times in a second.

once we changed it to schedule alert, problem was solved.

Re: Too many search jobs found in the dispatch directory

enno — Wed, 23 Mar 2016 23:25:38 GMT

My user base is constantly doing this... we seem to have to do a quarterly sweep of users building alerts out f real-time searches which then flood the dispatch directory over time...

Re: Too many search jobs found in the dispatch directory

CevSplunk — Thu, 28 Apr 2016 09:36:37 GMT

I confirm an alert storm that causes saturation splunk server, Once the alert removes the problem is set,

Re: Too many search jobs found in the dispatch directory

arunsundaram — Fri, 16 Jun 2017 13:22:32 GMT

Yes Dispatch directory can be cleared it wont cause any issues

Re: Too many search jobs found in the dispatch directory

nick405060 — Fri, 10 Aug 2018 17:49:17 GMT

Real-time alerts spammed our dispatch folder and ended up breaking the entire Splunk interface. Cleared /var/run/splunk/dispatch and modded the real-time alerts and boom, fixed.

If anyone doesn't know cron schedules, setting to "* * * * *" should fix this problem. It's alerting every minute instead of real-time.