Saved Searches are failing with error

[sanjeev543]

Hi All,
Recently I have noticed that some of the our Saved Searches are failing with the errors like below,

 "Failed to start search for id="scheduler__abcde__Qk1TX1dNX0lOVEdfTUVUUklDUw__RMD57438a1f3bbe5dac6_at_1588593600_88844". Dropping failedtostart token at path=/opt/splunk/var/run/splunk/dispatch/scheduler__abcde_Qk1TX1dNX0lOVEdfTUVUUklDUw__RMD57438a1f3bbe5dac6_at_1588593600_88844 to expedite dispatch cleanup

Could anyone suggest what could be the issue ?

[woodcock]

Open a support ticket and send them a diag.

[codebuilder]

I suspect @sanjeev543 is correct, but you can verify by running your search, wait for it to complete, then go to Job > Inspect Job then click on the search.log link.

Examine the entries in that log file and it should tell you exactly what the issue is.

If you do need to clean up the dispatch directory you can use the following:

/opt/splunk/bin/splunk cmd splunkd clean-dispatch /opt/splunk/var/run/splunk/old-dispatch-jobs/ -7d

This will move search artifacts to a new directory rather than deleting them. You'll need to create the directory first, and replace "-7d" with the value of your choice (7d = 7 days in this example).

[sanjeev543]

@codebuilder I don't see any files older than 2 days in dispatch directory , below is the confirmation from the command

Using logging configuration at /SplunkSHEBS/splunk/etc/log-cmdline.cfg.
dispatch dir:      /SplunkSHEBS/splunk/var/run/splunk/dispatch
destination dir:   /SplunkSHEBS/splunk/var/run/splunk/old-dispatch-jobs/
earliest mod time: 2020-04-29T03:32:03.000-04:00

total: 1331, moved: 0, failed: 0, remaining: 1331 job directories from /SplunkSHEBS/splunk/var/run/splunk/dispatch to /SplunkSHEBS/splunk/var/run/splunk/old-dispatch-jobs

/

Also when I use the sid to view the job properties, I don't see the job exists , even if I am searching for the job that was finished a couple of minutes ago and when I run the search query, I don't see any errors

Please suggest @woodcock @somesoni2 @MuS @martin_mueller

[codebuilder]

Is the directory full? Try running: df -h /SplunkSHEBS

[rkyadav]

@sanjeev543 ,

It looks like your dispatch directory is full and asking you to cleanup some.

You can navigate to /var/opt/splunk/var/run/splunk/dispatch for cleanup old files one from directories

[sanjeev543]

@rkyadav I didn't see the error saying dispatch directory is full and also I have seen above mentioned error trowing for only one Saved Search

[inawaz123]

@sanjeev543  have you resolved this issue ? i m seeing this issue in 8.0.3 search head cluster as well. If you have resolved this issue, can you please post your fix