Reporting

Too many search jobs found in the dispatch directory (found=2127, warning level=2000). This could negatively impact Splunk's performance, consider removing some of the old search jobs.

Communicator

I have read :

http://splunk-base.splunk.com/answers/29551/too-many-search-jobs-found-in-the-dispatch-directory

Yet the problem will not go away. Our dispatch directory is at :

[(prod) root@splunksearch01.prod.ostk.com ~]# ls -l /opt/splunk/var/run/splunk/dispatch
total 0
[(prod) root@splunksearch01.prod.ostk.com ~]#

We still have cleared out the files from both dispatch and dispatchtmp, but still continue to get the message. Running the clean-dispatch gives :

[(prod) root@splunksearch01.prod.ostk.com ~]# /opt/splunk/bin/splunk cmd splunkd clean-dispatch /opt/splunk/var/run/splunk/dispatch
...... (LOTS of these) .....
Could not move /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch/scheduler_nobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1345788000_8c88119f3789ab7b to /opt/splunk/var/run/splunk/dispatch/schedulernobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1345788000_8c88119f3789ab7b. Invalid cross-device link
Could not move /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch/scheduler
nobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1346392800_b2605fff19a01988 to /opt/splunk/var/run/splunk/dispatch/schedulernobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1346392800_b2605fff19a01988. Invalid cross-device link
Could not move /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch/scheduler
nobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1342159200_6428ee7431b0fba6 to /opt/splunk/var/run/splunk/dispatch/schedulernobody_search_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1342159200_6428ee7431b0fba6. Invalid cross-device link
total: 2199, moved: 0, failed: 200, remaining: 2199 job directories from /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch to /opt/splunk/var/run/splunk/dispatch
[(prod) root@splunksearch01.prod.ostk.com ~]#

It looks like it tells me we have 2199 jobs, but I can't find them anywhere. Ideas?

Tags (1)
0 Karma
1 Solution

Explorer

nocostk, are you running splunk with search head pooling? I only asked this based on your output:

Could not move /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch/schedulernobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1342159200_6428ee7431b0fba6 to /opt/splunk/var/run/splunk/dispatch/schedulernobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1342159200_6428ee7431b0fba6. Invalid cross-device link

When in pooled mode, all data shared between your search heads will always be in the NFS directory in this case:
/splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch

check out that directory and see if your search artifacts are there.

If you're constantly running into this issue, you could also increase the warning limit in limits.conf:

[search]
dispatch_dir_warning_size = 3000

View solution in original post

Explorer

nocostk, are you running splunk with search head pooling? I only asked this based on your output:

Could not move /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch/schedulernobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1342159200_6428ee7431b0fba6 to /opt/splunk/var/run/splunk/dispatch/schedulernobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1342159200_6428ee7431b0fba6. Invalid cross-device link

When in pooled mode, all data shared between your search heads will always be in the NFS directory in this case:
/splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch

check out that directory and see if your search artifacts are there.

If you're constantly running into this issue, you could also increase the warning limit in limits.conf:

[search]
dispatch_dir_warning_size = 3000

View solution in original post

Ultra Champion

I found that my shared pooling dispatch location had many empty dirs. I started by just removing those with a simple rmdir (because without params it would leave anywith nonempty and not a dir). That removed about 3k items for me.

0 Karma

Explorer

this will find out how many files you ahve that are over 30 days old

find /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch -type d -mtime +30 | wc -l

then you can do:
find /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch -type d -mtime +30 | xargs rm -rf

which will delete anything that hasnt been modified in the last 30 days, you'll have to run this with an account that has privileges to delete off your NFS store.

0 Karma

Communicator

That looks like it. 13,000 in there today, and it is pretty close to what is in the banner now. Looks like we have a few intense searches going on.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!