Reporting
Highlighted

Re: Too many search jobs found in the dispatch directory

Champion

Our issue looks like a bug in jobs. Certain jobs were not showing as completed and therefore hung around on disk until they expired. Issue was reported to Splunk Support and no word as to whether they will mark this as a bug or not. For now we have deleted all these old scheduled searches (probably some configuration issue after upgrade) and recreated them by hand.

Highlighted

Re: Too many search jobs found in the dispatch directory

Communicator

to add answer to the ellen,

some times, it won't delete because of <old_dispatch> directory what we created to move dispatch jobs may full. so create one more and move them. it works.

0 Karma
Highlighted

Re: Too many search jobs found in the dispatch directory

Communicator

might be "real time alerts" also a cause to produce many dispatch files.

for my case, one real time alert was triggering 3-5 times in a second.

once we changed it to schedule alert, problem was solved.

Highlighted

Re: Too many search jobs found in the dispatch directory

Explorer

My user base is constantly doing this... we seem to have to do a quarterly sweep of users building alerts out f real-time searches which then flood the dispatch directory over time...

0 Karma
Highlighted

Re: Too many search jobs found in the dispatch directory

New Member

I confirm an alert storm that causes saturation splunk server, Once the alert removes the problem is set,

0 Karma
Highlighted

Re: Too many search jobs found in the dispatch directory

Motivator

Real-time alerts spammed our dispatch folder and ended up breaking the entire Splunk interface. Cleared /var/run/splunk/dispatch and modded the real-time alerts and boom, fixed.

If anyone doesn't know cron schedules, setting to "* * * * *" should fix this problem. It's alerting every minute instead of real-time.

Highlighted

Re: Too many search jobs found in the dispatch directory

New Member

Yes Dispatch directory can be cleared it wont cause any issues

0 Karma