throttle alert once per day

[marees123]

Hi All,

My application is for guest enrollment. So new guest keep enrolling to this application, and we want to get alert if total enrollment reaches more than 300  in a day. And data must be pulled thru query for a given day. Only one alert should be triggered per day. Alert should be triggered next day if the count reaches more than 300

Query is

index="web" (host="web*" OR host="web1*") earliest = @d | search "Registered user tag: TAG id"| chart count(GUESTS_ENROLLED) as No_Of_Guests_Enrolled_in_a_day | where No_Of_Guests_Enrolled_in_a_day >=300

Requirement is

1. Value should be searched from beginning of the day 12.01 AM
2. email alert should be triggered once the count reaches 300 or more than that (between 12.00 AM - 11.59 PM)
3. only one email alert should be triggered per day
4. please advise what should be the search frequency.. like 5 mins.. 15 mins..., because alert should be triggered once the count reaches 300 (it may be between 12.00 AM to 11.59 PM anytime) . if i schedule the search to run at 11.59 PM everyday, but, if the count reaches to 300 by 2.00 AM, 22 hours gap will be there. so please advise the search frequency time as well. I guess i'm unable to use "Throttle" in search settings.

Please help.

Thanks.

[fredchase]

I had this same problem and I have not found a good answer on the forums. I finally solved it, so for others who are struggling with how to throttle an alert until the next day, here is my solution.

First, don't tick the Throttle checkbox in Alerts. Instead, find out whether an alert has already been triggered for the day with a subsearch. Use the subsearch as a function that returns the value of the number of previously triggered alerts for the day. If you have not yet had an alert for the day, it will return 0, so if the alert conditions are true and alerts_for_the_day=0 then you fire the alert, otherwise no. Here is an example for checking for when you are over the license limit where the name of the alert is "License Limit Exceeded: Over 100% Usage":

| rest splunk_server=local /services/licenser/pools | rename title AS Pool | search [rest splunk_server=local /services/licenser/groups | search is_active=1 | eval stack_id=stack_ids | fields stack_id] | eval quota=if(isnull(effective_quota),quota,effective_quota) | eval "Percentage of daily license limit used"=round(used_bytes/quota*100,2) | eval "Alert time"=strftime(now(), "%T %Z") | eval alert_count_today=[search index=_internal sourcetype=scheduler thread_id=AlertNotifier* savedsearch_name="License Limit Exceeded: Over 100% Usage" earliest=@d | where alert_actions!="" | stats count | return($count)] | where 'Percentage of daily license limit used' > 100 and alert_count_today = 0 | fields "Alert time" "Percentage of daily license limit used"

Essentially here is the format:

 ---enter your basic alert conditions--- |

eval alert_count_today=[search index=_internal sourcetype=scheduler thread_id=AlertNotifier* savedsearch_name=---enter the name of your alert here--- earliest=@d |

where alert_actions!="" |

stats count |

return($count)] |

where ---your basic alert triggers are evaluated--- and alert_count_today = 0 |

fields ---list the fields you want displayed in the event---

The key to making this work is understanding the event that is created every time a scheduled alert is run. The first part of the subsearch finds the alert events. In every alert event Splunk creates a field "alert_actions". If the alert is not triggered the value of alert_actions is set to an empty string. Hence the expression 'where alert_actions!=""' will only find events where an alert actually did get triggered. The "count" variable will be "0" when no alert has been triggered so far in the day, otherwise it will be "1". So if we return the value of the count variable and set the variable alert_count_today equal to the returned result, we now know whether an alert has been triggered or not.

In the case of the 300 guests, you just need to insert "where No_Of_Guests_Enrolled_in_a_day>=300 and alert_count_today=0" towards the end of the search string along with the initial search string, saved_search_name, and output fields.

[bill_kirby]

Hey, fredchase - this is a cool idea. I haven't had a chance to try it out yet, but just wanted to give you props.

[bill_kirby]

This looks like it's going to work for me. Thanks again, fredchase!

[jplumsdaine22 · Accepted solution]

I would do the following

1. Change chart to streamstats

   index="web" (host="web*" OR host="web1*") earliest = @d "Registered user tag: TAG id"| streamstats count(GUESTS_ENROLLED) as No_Of_Guests_Enrolled_in_a_day | where No_Of_Guests_Enrolled_in_a_day=300

This way your alert can also contain info on who the 300th guest , so you can fold their towels into swans or something

1. Trigger Condition should be Number of Results is Greater than 0

2. Under Action Options in the Edit Alert section, set 'When triggered, execute actions" to "For each result"

3. Tick the Throttle box

4. In "Suppress results containing field value", enter date_wday

5. In "Suppress triggering for" enter 24 hours

Your events should contain the date_wday field if your timestamps are correctly configured. The value of this field is the name of the day of the week. By using this throttling method, the alert will trigger a single time per day in a 24 hour period. at 12:00AM your date_wday field will change to the next day, so even if you reached 300 guests at 11:59PM the alert will fire for the next set of guests the following day

[marees123]

thanks a lot.

its working perfect....

thanks...

[jplumsdaine22]

No problem - any chance you can change your title to include the phrase 'throttle alert once per day' and add the tags alert and trigger?

[marees123]

yes. changed as suggested. thanks.

[jplumsdaine22]

I would set the schedule for how soon you want to know about the alert. For example, you could run the search in real time.

Have a read through http://docs.splunk.com/Documentation/Splunk/latest/Alert/Aboutalerts

Also have a look at the "Advanced conditional alert section here" http://docs.splunk.com/Documentation/Splunk/latest/Alert/Definescheduledalerts

[marees123]

thank you. yes getting alert...

but, one doubt...

i setup throttle value as 1 day, after triggering an email alert.

consider, if count reaches to 300 at 11.00 PM 23 dec 2015 and i got a mail alert. throttle value is 1 day, it means alert wont be triggered by next day, ie 24 dec 11.00 PM..

Again, next day, count reaches to 300 by 2.00 AM, 24 dec, will i get email alert? or due to that throttle value alert will be suppressed...

i guess, i'm not confusing you...

[jplumsdaine22]

If you only want to be alerted once, instead of using throttling, just use where No_Of_Guests_Enrolled_in_a_day =300 . That way you only get alerted once (Im assuming that value doesn't go down during a day)

[marees123]

thanks for your suggestion. Throttle didnt work. it is not for a day. it is taking for 24 hrs.

yes, we can try where No_Of_Guests_Enrolled_in_a_day =300.

but drawback are

if the count increases from 300 to higher, between the search duration (15 minutes), we will not get alert.
if we use where No_Of_Guests_Enrolled_in_a_day >=300, we will get multiple alerts...

any idea...?