Our issue looks like a bug in jobs. Certain jobs were not showing as completed and therefore hung around on disk until they expired. Issue was reported to Splunk Support and no word as to whether they will mark this as a bug or not. For now we have deleted all these old scheduled searches (probably some configuration issue after upgrade) and recreated them by hand.
to add answer to the ellen,
some times, it won't delete because of <old_dispatch> directory what we created to move dispatch jobs may full. so create one more and move them. it works.
might be "real time alerts" also a cause to produce many dispatch files.
for my case, one real time alert was triggering 3-5 times in a second.
once we changed it to schedule alert, problem was solved.
My user base is constantly doing this... we seem to have to do a quarterly sweep of users building alerts out f real-time searches which then flood the dispatch directory over time...
I confirm an alert storm that causes saturation splunk server, Once the alert removes the problem is set,
Real-time alerts spammed our dispatch folder and ended up breaking the entire Splunk interface. Cleared /var/run/splunk/dispatch and modded the real-time alerts and boom, fixed.
If anyone doesn't know cron schedules, setting to "* * * * *" should fix this problem. It's alerting every minute instead of real-time.