Reporting

Trigger alert when saved search has error

BernardEAI
Communicator

We have a number of saved searches (configured as alerts) that make use of custom search commands we wrote. It can happen that those custom commands fail to execute. This would cause the saved search to fail with an error message, which can be seen if you would run the search in the Splunk UI.

We would like alerts to be triggered if this happens to the saved search. In the Alert configuration, this type of trigger is not an option. 

Is there any way we can get an alert triggered when one of our saved searches fails with an error?

PS: we are running our app on a multi-tenant platform, so we do not have access to the internal logs, thus we cannot run a search like:

index=_internal sourcetype=scheduler status!=success 
| table _time search_type status user app savedsearch_name 

 

Labels (2)
0 Karma
1 Solution

BernardEAI
Communicator

@venkatasri thanks for the suggestion. 

I have tried getting the status of the saved search using an API call. I have run this API:

/servicesNS/admin/<app_name>/saved/searches/<save_search_name>/history

This returns a lot of the info on the search, but not the status!?!? I would have expected this would be in the data. Any idea if there is another endpoint that I can use that would have the status info?

UPDATE:

I managed to find an API call that would report on failed scheduled searches:

| rest /servicesNS/-/<app_name>/search/jobs | search isFailed=1
| table label, published, messages.fatal

This will work on our multi-tenant platform.

View solution in original post

0 Karma

venkatasri
SplunkTrust
SplunkTrust

Hi @BernardEAI 

Did you try Rest API's provided by Splunk if you got access to API's?

0 Karma

BernardEAI
Communicator

@venkatasri thanks for the suggestion. 

I have tried getting the status of the saved search using an API call. I have run this API:

/servicesNS/admin/<app_name>/saved/searches/<save_search_name>/history

This returns a lot of the info on the search, but not the status!?!? I would have expected this would be in the data. Any idea if there is another endpoint that I can use that would have the status info?

UPDATE:

I managed to find an API call that would report on failed scheduled searches:

| rest /servicesNS/-/<app_name>/search/jobs | search isFailed=1
| table label, published, messages.fatal

This will work on our multi-tenant platform.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...