How can i create notable events alert if any of correlation searches is getting skipped?
Try the following:
index=_internal sourcetype=scheduler status=skipped
| stats values(reason) as reason, count by savedsearch_name
When you run the search, let it execute, then click on "Save As" on the top right hand corner, then click on save as alert, fill in the details in the dialogue box which is pretty straight forward (If you want all results in one email, select in the dialogue box Once, if you want an individual email for each search, then select for each results) and then select the alert action as per your requirement. Ex: Send email alert action to send an email to you and others.
Hope this helps.
This is good, but it will return all skipped searches, not just correlation searches. Since the Scheduler log does not distinguish CSs from ordinary scheduled searches, we need to filter by CS name. We can get a list of all CS names using REST.
index=_internal sourcetype=scheduler status=skipped host=<<my SH>>
[ | rest /services/saved/searches splunk_server=local
| search is_scheduled=1 disabled=0 action.correlationsearch.enabled=1
| fields title
| rename title as savedsearch_name
| format ]
This query is not working i cant see any results
Do you have access to the _internal index? If not, you'll get no results. Did you replace the placeholder following "host="? Is the time range large enough to find skipped searches?
Have you tried running the subsearch by itself to verify it returns results?
The Monitoring Console has a search for skipped searches. See Search->Scheduler Activity. Use that search as a model to create a CS that detects skipped searches.