Reporting

Too many search jobs found in the dispatch directory (found=2127, warning level=2000). This could negatively impact Splunk's performance, consider removing some of the old search jobs.

nocostk
Communicator

I have read :

http://splunk-base.splunk.com/answers/29551/too-many-search-jobs-found-in-the-dispatch-directory

Yet the problem will not go away. Our dispatch directory is at :

[(prod) root@splunksearch01.prod.ostk.com ~]# ls -l /opt/splunk/var/run/splunk/dispatch
total 0
[(prod) root@splunksearch01.prod.ostk.com ~]#

We still have cleared out the files from both dispatch and dispatchtmp, but still continue to get the message. Running the clean-dispatch gives :

[(prod) root@splunksearch01.prod.ostk.com ~]# /opt/splunk/bin/splunk cmd splunkd clean-dispatch /opt/splunk/var/run/splunk/dispatch
...... (LOTS of these) .....
Could not move /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch/scheduler_nobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1345788000_8c88119f3789ab7b to /opt/splunk/var/run/splunk/dispatch/schedulernobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1345788000_8c88119f3789ab7b. Invalid cross-device link
Could not move /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch/scheduler
nobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1346392800_b2605fff19a01988 to /opt/splunk/var/run/splunk/dispatch/schedulernobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1346392800_b2605fff19a01988. Invalid cross-device link
Could not move /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch/scheduler
nobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1342159200_6428ee7431b0fba6 to /opt/splunk/var/run/splunk/dispatch/schedulernobody_search_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1342159200_6428ee7431b0fba6. Invalid cross-device link
total: 2199, moved: 0, failed: 200, remaining: 2199 job directories from /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch to /opt/splunk/var/run/splunk/dispatch
[(prod) root@splunksearch01.prod.ostk.com ~]#

It looks like it tells me we have 2199 jobs, but I can't find them anywhere. Ideas?

Tags (1)
0 Karma
1 Solution

seanwong
Explorer

nocostk, are you running splunk with search head pooling? I only asked this based on your output:

Could not move /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch/schedulernobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1342159200_6428ee7431b0fba6 to /opt/splunk/var/run/splunk/dispatch/schedulernobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1342159200_6428ee7431b0fba6. Invalid cross-device link

When in pooled mode, all data shared between your search heads will always be in the NFS directory in this case:
/splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch

check out that directory and see if your search artifacts are there.

If you're constantly running into this issue, you could also increase the warning limit in limits.conf:

[search]
dispatch_dir_warning_size = 3000

View solution in original post

seanwong
Explorer

nocostk, are you running splunk with search head pooling? I only asked this based on your output:

Could not move /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch/schedulernobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1342159200_6428ee7431b0fba6 to /opt/splunk/var/run/splunk/dispatch/schedulernobodysearch_Q2FycmllckludGVncmF0aW9uIEludmFsaWQgUHJvZHVjdCBXZWlnaHQ_at_1342159200_6428ee7431b0fba6. Invalid cross-device link

When in pooled mode, all data shared between your search heads will always be in the NFS directory in this case:
/splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch

check out that directory and see if your search artifacts are there.

If you're constantly running into this issue, you could also increase the warning limit in limits.conf:

[search]
dispatch_dir_warning_size = 3000

sloshburch
Splunk Employee
Splunk Employee

I found that my shared pooling dispatch location had many empty dirs. I started by just removing those with a simple rmdir (because without params it would leave anywith nonempty and not a dir). That removed about 3k items for me.

0 Karma

seanwong
Explorer

this will find out how many files you ahve that are over 30 days old

find /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch -type d -mtime +30 | wc -l

then you can do:
find /splunkconfig/splunk4.3.3/pooling/var/run/splunk/dispatch -type d -mtime +30 | xargs rm -rf

which will delete anything that hasnt been modified in the last 30 days, you'll have to run this with an account that has privileges to delete off your NFS store.

0 Karma

nocostk
Communicator

That looks like it. 13,000 in there today, and it is pretty close to what is in the banner now. Looks like we have a few intense searches going on.

0 Karma
Get Updates on the Splunk Community!

What's New in Splunk Enterprise 9.4: Features to Power Your Digital Resilience

Hey Splunky People! We are excited to share the latest updates in Splunk Enterprise 9.4. In this release we ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...