Alerting

Saved Searches are failing with error

sanjeev543
Communicator

Hi All,
Recently I have noticed that some of the our Saved Searches are failing with the errors like below,

 "Failed to start search for id="scheduler__abcde__Qk1TX1dNX0lOVEdfTUVUUklDUw__RMD57438a1f3bbe5dac6_at_1588593600_88844". Dropping failedtostart token at path=/opt/splunk/var/run/splunk/dispatch/scheduler__abcde_Qk1TX1dNX0lOVEdfTUVUUklDUw__RMD57438a1f3bbe5dac6_at_1588593600_88844 to expedite dispatch cleanup

Could anyone suggest what could be the issue ?

Labels (1)
0 Karma

woodcock
Esteemed Legend

Open a support ticket and send them a diag.

0 Karma

codebuilder
SplunkTrust
SplunkTrust

I suspect @sanjeev543 is correct, but you can verify by running your search, wait for it to complete, then go to Job > Inspect Job then click on the search.log link.

Examine the entries in that log file and it should tell you exactly what the issue is.

If you do need to clean up the dispatch directory you can use the following:

/opt/splunk/bin/splunk cmd splunkd clean-dispatch /opt/splunk/var/run/splunk/old-dispatch-jobs/ -7d

This will move search artifacts to a new directory rather than deleting them. You'll need to create the directory first, and replace "-7d" with the value of your choice (7d = 7 days in this example).

----
An upvote would be appreciated and Accept Solution if it helps!

sanjeev543
Communicator

@codebuilder I don't see any files older than 2 days in dispatch directory , below is the confirmation from the command

Using logging configuration at /SplunkSHEBS/splunk/etc/log-cmdline.cfg.
dispatch dir:      /SplunkSHEBS/splunk/var/run/splunk/dispatch
destination dir:   /SplunkSHEBS/splunk/var/run/splunk/old-dispatch-jobs/
earliest mod time: 2020-04-29T03:32:03.000-04:00

total: 1331, moved: 0, failed: 0, remaining: 1331 job directories from /SplunkSHEBS/splunk/var/run/splunk/dispatch to /SplunkSHEBS/splunk/var/run/splunk/old-dispatch-jobs

/

Also when I use the sid to view the job properties, I don't see the job exists , even if I am searching for the job that was finished a couple of minutes ago and when I run the search query, I don't see any errors

Please suggest @woodcock @somesoni2 @MuS @martin_mueller

0 Karma

codebuilder
SplunkTrust
SplunkTrust

Is the directory full? Try running: df -h /SplunkSHEBS

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

rkyadav
Path Finder

@sanjeev543 ,

It looks like your dispatch directory is full and asking you to cleanup some.

You can navigate to /var/opt/splunk/var/run/splunk/dispatch for cleanup old files one from directories

sanjeev543
Communicator

@rkyadav I didn't see the error saying dispatch directory is full and also I have seen above mentioned error trowing for only one Saved Search

0 Karma

inawaz123
Loves-to-Learn

@sanjeev543  have you resolved this issue ? i m seeing this issue in 8.0.3 search head cluster as well. If you have resolved this issue, can you please post your fix 

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...