Where can I find documentation for splunkd clean-dispatch command?

[the_wolverine]

./splunk cmd splunkd clean-dispatch

Where can I find the full documentation for this command which is used to "clean up" dispatch directory based on age of the directories?

[brianpaulsuther]

I am on Windows Server 2012.
You may have to find your dispatch folder , in my case here 😧 splunk var run splunk dispatch .
And manually delete directories or move them to your old-dispatch-jobs folder (you need to create that).
Because the CLI did not delete quite a few of mine.
After I manually deleted 'miraculously' SPLUNK started to render searches and dashboards correctly again.
At this stage I would guess that my creation of real time alert yesterday caused the issue but unclear why.

Hope this helps someone.
ps - Is there a document on good housekeeping for SPLUNK ?

[Ellen · Accepted solution]

If you run $SPLUNK_HOME/bin/splunk cmd splunkd clean-dispatch help
this will provided the usage information:

Sample from Splunk 6.1.4

$SPLUNK_HOME/bin/splunk cmd splunkd clean-dispatch help   
Use this command to move jobs whose last modification time is earlier than the specified time from the dispatch directory to the specified destination directory.   
usage: splunkd clean-dispatch '<destination directory where to move jobs>' '<latest job mod time>' 
example: splunkd clean-dispatch /tmp/old-dispatch-jobs/ -1month   
example: splunkd clean-dispatch /tmp/old-dispatch-jobs/ -10d@d   
example: splunkd clean-dispatch /tmp/old-dispatch-jobs/ 2011-06-01T12:34:56.000-07:00   

[joshiro]

We need to clean the dispatch directory in a SH clustered environment.

We didnt found any best practices for the clean-dispatch command and the Splunk documentation doesnt help either.
https://docs.splunk.com/Documentation/Splunk/9.0.3/Search/Dispatchdirectoryandsearchartifacts

Should we run the clean-dispatch command node per node? Stop node, clean-dispatch, start node?
Or should we stop the whole SH cluster, then clean-dispatch each node, and then start the nodes?

[cmeo]

This is most unsatisfactory in environments where access to the command line is restricted. It should be moved into Splunk Web.

Also, routine cleanup doesn't seem to work all that well with search head pooling. I keep seeing errors like this:

 Failed to reap \\svvaufs.DOMAIN.COM\SplunkPnV\var\run\splunk\dispatch\SummaryDirector_1427239891.2615.SERVER0081 because of The directory is not empty.

And I frequently see errors from Apps like Lookup Editor when trying to update files. These are not persistent, and the permissions are all set correctly before anyone asks--we had a huge go-round sorting this out with the Windows admins.

Nevertheless Search Head Pooling throws these errors intermittently and I often have to retry a few times to make changes stick.

[jplumsdaine22]

There are a number of actions that require the command line for splunk - it will be very difficult if you are in an environment with no command line access at all. You will not be able to manage splunk as a cluster for example.

[jkat54]

Sounds like terrible nfs latency. Try pinging your NFS from all search heads and see if one has packet loss / high latency.

[the_wolverine]

The documentation apparently doesn't exist as of this date. From what I have been able to figure out:

Syntax:
~/splunk/bin/splunk cmd splunkd clean-dispatch /tmp -24h@h

/tmp = the directory where you want the dispatch artifacts to be copied to.
-24h@h = the age when older dispatch artifacts are moved out of dispatch