Deployment Architecture

"Too many search jobs found in the dispatch directory" - Can we run this command on our clustered search heads to clean it?

sim_tcr
Communicator

We are on search head clustering with 4 search heads and version 6.3.3.
Recently started seeing WARNING:

Too many search jobs found in the dispatch directory (found=3186, warning level=2000). This could negatively impact Splunk's performance, consider removing some of the old search jobs.

Can we run below command on all our search heads to clean it?

splunk cmd splunkd clean-dispatch /apps/old-splunk_dispatch -1d

Will this only clean 1 day old jobs, or can we safely run it with -7d?

0 Karma

sowings
Splunk Employee
Splunk Employee

While it's not a direct answer to your question, consider WHY these jobs may be piling up. See my post on this answer for more information.

Regarding @SloshBurch's comment: if the jobs are adhoc in nature (the SID begins with a 10-digit number), then cleaning from a single member of the SHC will be OK for that member. There's no replication of adhoc search jobs. If, however, the pile up of dispatch directories is from scheduled search activity, it might make sense to evaluate the TTLs of those jobs to understand why they're piling up. The threshold of 2000 entries is a warning level, but given the way SHC replicates dispatch directories, it might be a bit low. If you're routinely up in the 3000 range, it may be reasonable (with the guidance of support) to raise the configured limit.

0 Karma

sloshburch
Ultra Champion

sim_tcr
Communicator

I ran the command only few got moved. Getting below error on some

Could not move /apps/splunk/var/run/splunk/dispatch/_splunktemps to /apps/old-splunk_dispatch/_splunktemps. Directory not empty
total: 2014, moved: 0, failed: 1, remaining: 2014 job directories from /apps/splunk/var/run/splunk/dispatch to /apps/old-splunk_dispatch
0 Karma

sloshburch
Ultra Champion

My guess is that the "Directory not empty" was similar to a folder currently being used (open file handles). Would you confirm that you stopped that host's splunk instance - just in case it was working on the FS at the same time? I'm also wondering if cleaning dispatch with a SHC will be sufficient at cleaning the other SHC members as well (since file system changes in a SHC are not synchronized).

0 Karma

ddrillic
Ultra Champion
0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!