Other Usage

Get list of failed or skipped scheduled saved searches


Hi All,

I'm trying to make a report of failed or skipped scheduled saved searches.
The report should include the list of scheduled saved searches which failed or skipped when,
- splunk is down due to some issues
- during the splunk retstart / maintenance
- splunk query issue

index=_internal source=*scheduler.log search_type=scheduled

only gives the list of scheduled saved searches, which were skipped but not failed

are there any configuration that can execute these saved searches when Splunk is restarted or back to normal function.

let me know if any other details are required.


0 Karma


This SPL will give you the failed saved searches:

index=_audit sourcetype=audittrail TERM(action=search) (TERM(info=bad_request)) (TERM(search=*) OR TERM(savedsearch=*)) NOT (MongoModificationsTracker OR (INFO (metrics OR PeriodicHealthReporter OR LicenseUsage) OR StreamedSearch) OR TERM(info=granted) OR (TERM(info=completed) TERM(has_error_warn=false) TERM(fully_completed_search=true)) OR GET ) provenance=scheduler
| rex mode=sed field=search "s/^'//"
| rex mode=sed field=search "s/'$//"
| rex mode=sed field=search_id "s/^'//"
| rex mode=sed field=search_id "s/'$//"
| table _time app info has_error_warn mode provenance savedsearch_name search search_id src user total_run_time

0 Karma

You could also use MC to look those. Just select MC -> Search -> Scheduler and there are couple of different dashboard. Then select suitable panel and open SPL for it and modify as needed.
0 Karma


| rest /servicesNS/-/-/search/jobs

delegate=scheduler indicates this is a scheduled search. Null value indicates that this was an ad-hoc search

dispatchState provides you the status of the search. dispatchState=Failed gives you all the Failed searches

This will give you all the search jobs which are visible in the Jobs manager. Expired searches are removed so if you need historical data, best to run this on a cadence and save to a summary search.

Hope this helps.


0 Karma


The results of my query and the your query are different even for failed/skipped saved searches.

According to rest query, there are very few failed searches, but when compared manually with searches that were not executed, than they were not in the results.

0 Karma


Hi @gaurav_maniar

The rest endpoint only provides the most recent searches (searches whose ttl hasn't expired) and does not maintain history; requiring that this be saved to a summary index to maintain history. If the search is visible when you click on Actvity>Jobs, then these will be visible in the rest endpoint.

What time range did you run the query against the internal index for?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...