Alerting

How do I suppress alerts until the next day at 12 am and not 24 hours?

New Member

I have a couple of alerts for License usage set to check every hour when they exceed 75 %. At the moment, I receive an email every hour after it reaches the threshold. I only want to be notified one time for that day. If I use the throttle option and suppress for 1 day, it will wait 24 hours until it alerts me again. This is an issue because if the threshold is exceeded earlier the following day, I won't get notified until that time runs out. I would like to suppress the alert until 12 am that night. Can anyone please help? Thank you

0 Karma

Explorer

I posted a solution to this general problem here: https://answers.splunk.com/answers/337985/throttle-alert-once-per-day.html
In short, the search string I used to trigger once a day alerting when over the license limit is this:

| rest splunk_server=local /services/licenser/pools | rename title AS Pool | search [rest splunk_server=local /services/licenser/groups | search is_active=1 | eval stack_id=stack_ids | fields stack_id] | eval quota=if(isnull(effective_quota),quota,effective_quota) | eval "Percentage of daily license limit used"=round(used_bytes/quota*100,2) | eval "Alert time"=strftime(now(), "%T %Z") | eval alert_count_today=[search index=_internal sourcetype=scheduler thread_id=AlertNotifier* savedsearch_name="License Limit Exceeded: Over 100% Usage" earliest=@d | where alert_actions!="" | stats count | return($count)] | where 'Percentage of daily license limit used' > 100 and alert_count_today = 0 | fields "Alert time" "Percentage of daily license limit used"

For more details see the original post.

Revered Legend

Try this as your alert search (no throttling needs to be set)

| rest /services/licenser/usage | eval "% used"=round(slaves_usage_bytes/quota*100,2) | appendcols [search index=_internal sourcetype=scheduler thread_id=AlertNotifier* savedsearch_name="PUTYOURALERTSEARCHNAMEHERE" earliest=@d | head 1 | table _time]  | where '% used' > 75 AND isnull(_time)| fields "% used", "updated" 

The appendcols subsearch will try to find the timestamp of an alert that was fired today. If it finds one, the alert will be be fired.

New Member

Thank you, I inserted the search and will test today. I sent ahead and removed any throttle in the Trigger Condition screen.

0 Karma

Path Finder

It was able to alert me based on the threshold I set. However, it continues to send an alert. How would you schedule this report in the Type and Trigger condition?

0 Karma

Revered Legend

What is your alert search ?

0 Karma

New Member
| rest /services/licenser/usage | eval "% used"=round(slaves_usage_bytes/quota*100,2) | where '% used' > 75 | fields "% used", "updated"
0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!