Splunk Search

Replicated scheduled search not removed- Can I know the Period of the scheduler search and where it is replicated from?

louismai
Path Finder

Hi,

I keep receiving the warning message related "Search peer xxxxxx03 has the following message: Dispatch Command: The number of search artifacts in the dispatch directory is higher than recommended (count=7948, warning threshold=5000) and could have an impact on search performance. Remove excess search artifacts using the "splunk clean-dispatch" CLI command, and review artifact retention policies in limits.conf and savedsearches.conf. You can also raise this warning threshold in limits.conf / dispatch_dir_warning_size".

I keep cleaning the that SH (other 3 SH don't have problems) dispatch folders, but the job increases very fast. I figured out that the dispatch folder has about 5000 records of rsa_scheduler. Many are more 2-3 hours old which are strange.

So how can I know the Period of the scheduler search and where it is replicated from?
For example:
drwx------. 2 splunk splunk 263 Sep 16 14:03 rsa_scheduler_nobodynmonRMD5ee48120c2dd6c8cc_at_1568606400_26400_546F2A6F-BFB1-4954-9173-74A67615D481
drwx------. 2 splunk splunk 363 Sep 16 14:03
rsa_scheduler
nobodyuberAgent_RMD5b4e9f6a64f89a433_at_1568561400_15572_54E1D115-8124-4FE4-A9EB-5B4AADB08D33

Tks.

Labels (1)
Tags (1)
0 Karma

joshiro
Communicator

Hi, we are having a similar issue, have you managed to solve it?

We need to clean the dispatch directory in a SH clustered environment.

We didnt found any best practices for the clean-dispatch command and the Splunk documentation doesnt help either.
https://docs.splunk.com/Documentation/Splunk/9.0.3/Search/Dispatchdirectoryandsearchartifacts

Should we run the clean-dispatch command node per node? Stop node, clean-dispatch, start node?
Or should we stop the whole SH cluster, then clean-dispatch each node, and then start the nodes?

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...