Get list of failed or skipped scheduled saved searches

[gaurav_maniar]

Hi All,

I'm trying to make a report of failed or skipped scheduled saved searches.
The report should include the list of scheduled saved searches which failed or skipped when,
- splunk is down due to some issues
- during the splunk retstart / maintenance
- splunk query issue

index=_internal source=*scheduler.log search_type=scheduled

only gives the list of scheduled saved searches, which were skipped but not failed

are there any configuration that can execute these saved searches when Splunk is restarted or back to normal function.

let me know if any other details are required.

Thanks.

[landen99_gdms]

This SPL will give you the failed saved searches:

index=_audit sourcetype=audittrail TERM(action=search) (TERM(info=bad_request)) (TERM(search=*) OR TERM(savedsearch=*)) NOT (MongoModificationsTracker OR (INFO (metrics OR PeriodicHealthReporter OR LicenseUsage) OR StreamedSearch) OR TERM(info=granted) OR (TERM(info=completed) TERM(has_error_warn=false) TERM(fully_completed_search=true)) OR GET ) provenance=scheduler
| rex mode=sed field=search "s/^'//"
| rex mode=sed field=search "s/'$//"
| rex mode=sed field=search_id "s/^'//"
| rex mode=sed field=search_id "s/'$//"
| table _time app info has_error_warn mode provenance savedsearch_name search search_id src user total_run_time

[isoutamo]

You could also use MC to look those. Just select MC -> Search -> Scheduler and there are couple of different dashboard. Then select suitable panel and open SPL for it and modify as needed.

[arjunpkishore5]

| rest /servicesNS/-/-/search/jobs

delegate=scheduler indicates this is a scheduled search. Null value indicates that this was an ad-hoc search

dispatchState provides you the status of the search. dispatchState=Failed gives you all the Failed searches

This will give you all the search jobs which are visible in the Jobs manager. Expired searches are removed so if you need historical data, best to run this on a cadence and save to a summary search.

Hope this helps.

Cheers

[gaurav_maniar]

The results of my query and the your query are different even for failed/skipped saved searches.

According to rest query, there are very few failed searches, but when compared manually with searches that were not executed, than they were not in the results.

[arjunpkishore5]

Hi @gaurav_maniar

The rest endpoint only provides the most recent searches (searches whose ttl hasn't expired) and does not maintain history; requiring that this be saved to a summary index to maintain history. If the search is visible when you click on Actvity>Jobs, then these will be visible in the rest endpoint.

What time range did you run the query against the internal index for?