Too many search jobs found in the dispatch directory (found=2127, warning level=2000). This could negatively impact Splunk's performance, consider removing some of the old search jobs.

[nocostk]

I have read :

http://splunk-base.splunk.com/answers/29551/too-many-search-jobs-found-in-the-dispatch-directory

Yet the problem will not go away. Our dispatch directory is at :

[(prod) root@splunksearch01.prod.ostk.com ~]# ls -l /opt/splunk/var/run/splunk/dispatch
total 0
[(prod) root@splunksearch01.prod.ostk.com ~]#

We still have cleared out the files from both dispatch and dispatchtmp, but still continue to get the message. Running the clean-dispatch gives :

[(prod) root@splunksearch01.prod.ostk.com ~]# /opt/splunk/bin/splunk cmd splunkd clean-dispatch /opt/splunk/var/run/splunk/dispatch
...... (LOTS of these) .....
Could not move /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch/scheduler_nobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1345788000_8c88119f3789ab7b to /opt/splunk/var/run/splunk/dispatch/schedulernobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1345788000_8c88119f3789ab7b. Invalid cross-device link
Could not move /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch/schedulernobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1346392800_b2605fff19a01988 to /opt/splunk/var/run/splunk/dispatch/schedulernobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1346392800_b2605fff19a01988. Invalid cross-device link
Could not move /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch/schedulernobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1342159200_6428ee7431b0fba6 to /opt/splunk/var/run/splunk/dispatch/schedulernobody_search_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1342159200_6428ee7431b0fba6. Invalid cross-device link
total: 2199, moved: 0, failed: 200, remaining: 2199 job directories from /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch to /opt/splunk/var/run/splunk/dispatch
[(prod) root@splunksearch01.prod.ostk.com ~]#

It looks like it tells me we have 2199 jobs, but I can't find them anywhere. Ideas?

[seanwong · Accepted solution]

nocostk, are you running splunk with search head pooling? I only asked this based on your output:

Could not move /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch/schedulernobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1342159200_6428ee7431b0fba6 to /opt/splunk/var/run/splunk/dispatch/schedulernobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1342159200_6428ee7431b0fba6. Invalid cross-device link

When in pooled mode, all data shared between your search heads will always be in the NFS directory in this case:
/splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch

check out that directory and see if your search artifacts are there.

If you're constantly running into this issue, you could also increase the warning limit in limits.conf:

[search]
dispatch_dir_warning_size = 3000

[sloshburch]

I found that my shared pooling dispatch location had many empty dirs. I started by just removing those with a simple rmdir (because without params it would leave anywith nonempty and not a dir). That removed about 3k items for me.

[seanwong]

this will find out how many files you ahve that are over 30 days old

find /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch -type d -mtime +30 | wc -l

then you can do:
find /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch -type d -mtime +30 | xargs rm -rf

which will delete anything that hasnt been modified in the last 30 days, you'll have to run this with an account that has privileges to delete off your NFS store.

[nocostk]

That looks like it. 13,000 in there today, and it is pretty close to what is in the banner now. Looks like we have a few intense searches going on.