Alerting

Saved Searches are failing with error

sanjeev543
Communicator

Hi All,
Recently I have noticed that some of the our Saved Searches are failing with the errors like below,

 "Failed to start search for id="scheduler__abcde__Qk1TX1dNX0lOVEdfTUVUUklDUw__RMD57438a1f3bbe5dac6_at_1588593600_88844". Dropping failedtostart token at path=/opt/splunk/var/run/splunk/dispatch/scheduler__abcde_Qk1TX1dNX0lOVEdfTUVUUklDUw__RMD57438a1f3bbe5dac6_at_1588593600_88844 to expedite dispatch cleanup

Could anyone suggest what could be the issue ?

Labels (1)
0 Karma

woodcock
Esteemed Legend

Open a support ticket and send them a diag.

0 Karma

codebuilder
Influencer

I suspect @sanjeev543 is correct, but you can verify by running your search, wait for it to complete, then go to Job > Inspect Job then click on the search.log link.

Examine the entries in that log file and it should tell you exactly what the issue is.

If you do need to clean up the dispatch directory you can use the following:

/opt/splunk/bin/splunk cmd splunkd clean-dispatch /opt/splunk/var/run/splunk/old-dispatch-jobs/ -7d

This will move search artifacts to a new directory rather than deleting them. You'll need to create the directory first, and replace "-7d" with the value of your choice (7d = 7 days in this example).

----
An upvote would be appreciated and Accept Solution if it helps!

sanjeev543
Communicator

@codebuilder I don't see any files older than 2 days in dispatch directory , below is the confirmation from the command

Using logging configuration at /SplunkSHEBS/splunk/etc/log-cmdline.cfg.
dispatch dir:      /SplunkSHEBS/splunk/var/run/splunk/dispatch
destination dir:   /SplunkSHEBS/splunk/var/run/splunk/old-dispatch-jobs/
earliest mod time: 2020-04-29T03:32:03.000-04:00

total: 1331, moved: 0, failed: 0, remaining: 1331 job directories from /SplunkSHEBS/splunk/var/run/splunk/dispatch to /SplunkSHEBS/splunk/var/run/splunk/old-dispatch-jobs

/

Also when I use the sid to view the job properties, I don't see the job exists , even if I am searching for the job that was finished a couple of minutes ago and when I run the search query, I don't see any errors

Please suggest @woodcock @somesoni2 @MuS @martin_mueller

0 Karma

codebuilder
Influencer

Is the directory full? Try running: df -h /SplunkSHEBS

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

rkyadav
Path Finder

@sanjeev543 ,

It looks like your dispatch directory is full and asking you to cleanup some.

You can navigate to /var/opt/splunk/var/run/splunk/dispatch for cleanup old files one from directories

sanjeev543
Communicator

@rkyadav I didn't see the error saying dispatch directory is full and also I have seen above mentioned error trowing for only one Saved Search

0 Karma

inawaz123
New Member

@sanjeev543  have you resolved this issue ? i m seeing this issue in 8.0.3 search head cluster as well. If you have resolved this issue, can you please post your fix 

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!