Trigger alert when saved search has error

[BernardEAI]

We have a number of saved searches (configured as alerts) that make use of custom search commands we wrote. It can happen that those custom commands fail to execute. This would cause the saved search to fail with an error message, which can be seen if you would run the search in the Splunk UI.

We would like alerts to be triggered if this happens to the saved search. In the Alert configuration, this type of trigger is not an option. 

Is there any way we can get an alert triggered when one of our saved searches fails with an error?

PS: we are running our app on a multi-tenant platform, so we do not have access to the internal logs, thus we cannot run a search like:

index=_internal sourcetype=scheduler status!=success 
| table _time search_type status user app savedsearch_name 

[venkatasri]

Hi @BernardEAI 

Did you try Rest API's provided by Splunk if you got access to API's?

[BernardEAI · Accepted solution]

@venkatasri thanks for the suggestion. 

I have tried getting the status of the saved search using an API call. I have run this API:

/servicesNS/admin/<app_name>/saved/searches/<save_search_name>/history

This returns a lot of the info on the search, but not the status!?!? I would have expected this would be in the data. Any idea if there is another endpoint that I can use that would have the status info?

UPDATE:

I managed to find an API call that would report on failed scheduled searches:

| rest /servicesNS/-/<app_name>/search/jobs | search isFailed=1
| table label, published, messages.fatal

This will work on our multi-tenant platform.