topic Re: Alert when "raises by" doesn't work in Alerting

Alert when "raises by" doesn't work

rocketboots_ser — Fri, 22 Feb 2019 07:23:59 GMT

I am trying to raise an alert when the number of results raises by 1. Each result represents a device going offline and I need to send an email every time a device goes offline. I have a scheduled search every 5 minutes because I want to remember the current state in case one device goes online and another one goes offline.

When I configured the alert I had 0 results. 10 minutes later one device went offline and then 10 minutes after another one went offline. I didn't receive any email. I'm not sure if I've misinterpreted the "rises by" setting, or it's not working. What are your thoughts?

Re: Alert when "raises by" doesn't work

lakshman239 — Fri, 22 Feb 2019 09:56:50 GMT

pls share your search

Re: Alert when "raises by" doesn't work

rocketboots_ser — Sun, 24 Feb 2019 22:44:12 GMT

| mcatalog values(device) as device where index=*_metric 
| mvexpand device
| join type=left device [
    | mstats max(_value) as qgets where metric_name=*.QueueGets, queue_name=_monitor, service=*nq index=*_metric earliest=-3m by device, customer
]
| eval online = if(qgets>0, "Online", "Offline")
| search online=Offline

Re: Alert when "raises by" doesn't work

woodcock — Wed, 06 Mar 2019 02:35:54 GMT

I never use the alert trigger features; I always implement my own directly in the SPL of the search itself. That way I can see the history of what is going on and debug things. The way that you do it is to save the previous results in lookup file by using | outputlookup YourSearchResultsHistoryLookupFile.csv Although the use case is different, the mechanics of what you need to do to go this route can be found in the sentinel example here:
https://conf.splunk.com/session/2015/conf2015-LookupTalk.pdf

Re: Alert when "raises by" doesn't work

dchoi_splunk — Wed, 30 Sep 2020 01:00:34 GMT

From the scheduler log:
If the permission of the Alert has been created as shared App or Globally, the scheduler fires the Alert as below:

[INFO SavedSplunker - savedsearch_id="nobody;search;your_alert_name“, search_type="scheduled", user="admin", app="search", savedsearch_name="your_alert_name"]

Then you will note that savedsearch_id="nobody;search;your_alert_name” does not match the user="admin", app="search", savedsearch_name=“your_alert_name”, here the specific user is "admin" vs "nobody"

In that case, you need to create the Alert as Private Or need to change the owner to "nobody" in local.meta manually if it’s created as App or Globally shared,
E.g under $SPLUNK_HOME/etc/apps/search/metadata/local.meta

From

[savedsearches/your_alert]
owner = admin

To
[savedsearches/your_alert]
owner = nobody

Then check whether the owner has been changed for the alert by clicking "Settings" -> "Searches, Reports, and Alerts" and check "Owner" field for the alert.

Hope it helps