topic Re: Alerts based on result range in Alerting

Alerts based on result range

prabhuanandampu — Thu, 20 Apr 2017 20:02:29 GMT

Hi,

I'm trying to set up alerts based on number of results we received from a search.

If the number of results returned is between x-y, need to send to a particular set of recipient
If the number of results returned is between (y+1)-z, need to send to a different set of recipient

But now, I'm not able to do this in the alert. Someone able to do this ?

Thanks
Prabhu

Re: Alerts based on result range

jonmargulies — Thu, 20 Apr 2017 20:34:53 GMT

I can think of two ways of doing this:
1) The easiest, but most inefficient, option is to set up two almost identical alerts. Both alerts would run the same search at the same time, but they'd have different actions. One alert would have an action with one range (x-y), and send to the first set of recipients, and the second alert would use the same query about have the other range ((y+1)-z), and send to the second set of recipients. The problem with this option, of course, is running twice as many searches as you need to.
2) Instead of an alert, create a saved search that uses the | sendemail command (that's basically what an alert is anyway). Basically what you can do is have an eval that sets a "to" field based on the ranges. Your query would end up looking roughly like ... | stats count | eval to=case(count>10, "address1, address2", count>5, "address3, address4", 1=1, "") | sendemail to="$result.to$" from="sender@example.net" subject="test" message="test2". If the to field is empty, (i.e., count<5), nothing will send.

Here's a Splunk Answers post that describes how to use sendemail this way: https://answers.splunk.com/answers/213340/how-to-get-splunk-sendemail-command-to-send-multip.html

A third option is basically a more efficient version of option 1: make a kvstore that tracks state, and use a saved search to populate that kvstore. Then create two alerts that just check that kvstore (which is basically a zero-cost query).

Re: Alerts based on result range

prabhuanandampu — Thu, 20 Apr 2017 22:27:37 GMT

Second option is something nice to try it. Unfortunately, we are posting the alerts to the slack channels. So, will need to look for alternatives for sendmail.

Re: Alerts based on result range

jonmargulies — Tue, 29 Sep 2020 13:46:44 GMT

Not a problem! If that's the case, there are two ways you can go:
1) Create a custom Python command sendslack that posts the results to Slack and use the same logic as Option 2 above.
2) Take a look at the third option I mentioned above. While it requires a scheduled search and two alerts, the two alerts are basically free in terms of performance/load. All you need is a very simply KVstore that has one field: last_count. So your main savedsearch would basically be ... | stats count | rename count | outputlookup alert_count_lookup. and each alert query would just be | inputlookup alert_count_lookup | search count>x count<y+1.

Re: Alerts based on result range

prabhuanandampu — Fri, 21 Apr 2017 17:38:51 GMT

Option 1 sounds interesting
https://www.splunk.com/blog/2014/04/14/building-custom-search-commands-in-python-part-i-a-simple-generating-command/

Will try that out