Reporting

Get list of failed or skipped scheduled saved searches

gaurav_maniar
Builder

Hi All,

I'm trying to make a report of failed or skipped scheduled saved searches.
The report should include the list of scheduled saved searches which failed or skipped when,
- splunk is down due to some issues
- during the splunk retstart / maintenance
- splunk query issue

index=_internal source=*scheduler.log search_type=scheduled

only gives the list of scheduled saved searches, which were skipped but not failed

are there any configuration that can execute these saved searches when Splunk is restarted or back to normal function.

let me know if any other details are required.

Thanks.

0 Karma

landen99_gdms
Engager

This SPL will give you the failed saved searches:

index=_audit sourcetype=audittrail TERM(action=search) (TERM(info=bad_request)) (TERM(search=*) OR TERM(savedsearch=*)) NOT (MongoModificationsTracker OR (INFO (metrics OR PeriodicHealthReporter OR LicenseUsage) OR StreamedSearch) OR TERM(info=granted) OR (TERM(info=completed) TERM(has_error_warn=false) TERM(fully_completed_search=true)) OR GET ) provenance=scheduler
| rex mode=sed field=search "s/^'//"
| rex mode=sed field=search "s/'$//"
| rex mode=sed field=search_id "s/^'//"
| rex mode=sed field=search_id "s/'$//"
| table _time app info has_error_warn mode provenance savedsearch_name search search_id src user total_run_time

0 Karma

isoutamo
SplunkTrust
SplunkTrust
You could also use MC to look those. Just select MC -> Search -> Scheduler and there are couple of different dashboard. Then select suitable panel and open SPL for it and modify as needed.
0 Karma

arjunpkishore5
Motivator

| rest /servicesNS/-/-/search/jobs

delegate=scheduler indicates this is a scheduled search. Null value indicates that this was an ad-hoc search

dispatchState provides you the status of the search. dispatchState=Failed gives you all the Failed searches

This will give you all the search jobs which are visible in the Jobs manager. Expired searches are removed so if you need historical data, best to run this on a cadence and save to a summary search.

Hope this helps.

Cheers

0 Karma

gaurav_maniar
Builder

The results of my query and the your query are different even for failed/skipped saved searches.

According to rest query, there are very few failed searches, but when compared manually with searches that were not executed, than they were not in the results.

0 Karma

arjunpkishore5
Motivator

Hi @gaurav_maniar

The rest endpoint only provides the most recent searches (searches whose ttl hasn't expired) and does not maintain history; requiring that this be saved to a summary index to maintain history. If the search is visible when you click on Actvity>Jobs, then these will be visible in the rest endpoint.

What time range did you run the query against the internal index for?

0 Karma
Get Updates on the Splunk Community!

Build Scalable Security While Moving to Cloud - Guide From Clayton Homes

 Clayton Homes faced the increased challenge of strengthening their security posture as they went through ...

Mission Control | Explore the latest release of Splunk Mission Control (2.3)

We’re happy to announce the release of Mission Control 2.3 which includes several new and exciting features ...

Cloud Platform | Migrating your Splunk Cloud deployment to Python 3.7

Python 2.7, the last release of Python 2, reached End of Life back on January 1, 2020. As part of our larger ...