Replicated scheduled search not removed- Can I know the Period of the scheduler search and where it is replicated from?

[louismai]

Hi,

I keep receiving the warning message related "Search peer xxxxxx03 has the following message: Dispatch Command: The number of search artifacts in the dispatch directory is higher than recommended (count=7948, warning threshold=5000) and could have an impact on search performance. Remove excess search artifacts using the "splunk clean-dispatch" CLI command, and review artifact retention policies in limits.conf and savedsearches.conf. You can also raise this warning threshold in limits.conf / dispatch_dir_warning_size".

I keep cleaning the that SH (other 3 SH don't have problems) dispatch folders, but the job increases very fast. I figured out that the dispatch folder has about 5000 records of rsa_scheduler. Many are more 2-3 hours old which are strange.

So how can I know the Period of the scheduler search and where it is replicated from?
For example:
drwx------. 2 splunk splunk 263 Sep 16 14:03 rsa_scheduler_nobodynmonRMD5ee48120c2dd6c8cc_at_1568606400_26400_546F2A6F-BFB1-4954-9173-74A67615D481
drwx------. 2 splunk splunk 363 Sep 16 14:03
rsa_schedulernobodyuberAgent_RMD5b4e9f6a64f89a433_at_1568561400_15572_54E1D115-8124-4FE4-A9EB-5B4AADB08D33

Tks.

[joshiro]

Hi, we are having a similar issue, have you managed to solve it?

We need to clean the dispatch directory in a SH clustered environment.

We didnt found any best practices for the clean-dispatch command and the Splunk documentation doesnt help either.
https://docs.splunk.com/Documentation/Splunk/9.0.3/Search/Dispatchdirectoryandsearchartifacts

Should we run the clean-dispatch command node per node? Stop node, clean-dispatch, start node?
Or should we stop the whole SH cluster, then clean-dispatch each node, and then start the nodes?