I'm trying to make a report of failed or skipped scheduled saved searches.
The report should include the list of scheduled saved searches which failed or skipped when,
- splunk is down due to some issues
- during the splunk retstart / maintenance
- splunk query issue
index=_internal source=*scheduler.log search_type=scheduled
only gives the list of scheduled saved searches, which were skipped but not failed
are there any configuration that can execute these saved searches when Splunk is restarted or back to normal function.
let me know if any other details are required.
This SPL will give you the failed saved searches:
index=_audit sourcetype=audittrail TERM(action=search) (TERM(info=bad_request)) (TERM(search=*) OR TERM(savedsearch=*)) NOT (MongoModificationsTracker OR (INFO (metrics OR PeriodicHealthReporter OR LicenseUsage) OR StreamedSearch) OR TERM(info=granted) OR (TERM(info=completed) TERM(has_error_warn=false) TERM(fully_completed_search=true)) OR GET ) provenance=scheduler
| rex mode=sed field=search "s/^'//"
| rex mode=sed field=search "s/'$//"
| rex mode=sed field=search_id "s/^'//"
| rex mode=sed field=search_id "s/'$//"
| table _time app info has_error_warn mode provenance savedsearch_name search search_id src user total_run_time
| rest /servicesNS/-/-/search/jobs
delegate=scheduler indicates this is a scheduled search. Null value indicates that this was an ad-hoc search
dispatchState provides you the status of the search. dispatchState=Failed gives you all the Failed searches
This will give you all the search jobs which are visible in the Jobs manager. Expired searches are removed so if you need historical data, best to run this on a cadence and save to a summary search.
Hope this helps.
The results of my query and the your query are different even for failed/skipped saved searches.
According to rest query, there are very few failed searches, but when compared manually with searches that were not executed, than they were not in the results.
The rest endpoint only provides the most recent searches (searches whose ttl hasn't expired) and does not maintain history; requiring that this be saved to a summary index to maintain history. If the search is visible when you click on Actvity>Jobs, then these will be visible in the rest endpoint.
What time range did you run the query against the internal index for?