Alerting

How do I suppress alerts until the next day at 12 am and not 24 hours?

splunkfmpa
New Member

I have a couple of alerts for License usage set to check every hour when they exceed 75 %. At the moment, I receive an email every hour after it reaches the threshold. I only want to be notified one time for that day. If I use the throttle option and suppress for 1 day, it will wait 24 hours until it alerts me again. This is an issue because if the threshold is exceeded earlier the following day, I won't get notified until that time runs out. I would like to suppress the alert until 12 am that night. Can anyone please help? Thank you

0 Karma

fredchase
Explorer

I posted a solution to this general problem here: https://answers.splunk.com/answers/337985/throttle-alert-once-per-day.html
In short, the search string I used to trigger once a day alerting when over the license limit is this:

| rest splunk_server=local /services/licenser/pools | rename title AS Pool | search [rest splunk_server=local /services/licenser/groups | search is_active=1 | eval stack_id=stack_ids | fields stack_id] | eval quota=if(isnull(effective_quota),quota,effective_quota) | eval "Percentage of daily license limit used"=round(used_bytes/quota*100,2) | eval "Alert time"=strftime(now(), "%T %Z") | eval alert_count_today=[search index=_internal sourcetype=scheduler thread_id=AlertNotifier* savedsearch_name="License Limit Exceeded: Over 100% Usage" earliest=@d | where alert_actions!="" | stats count | return($count)] | where 'Percentage of daily license limit used' > 100 and alert_count_today = 0 | fields "Alert time" "Percentage of daily license limit used"

For more details see the original post.

somesoni2
Revered Legend

Try this as your alert search (no throttling needs to be set)

| rest /services/licenser/usage | eval "% used"=round(slaves_usage_bytes/quota*100,2) | appendcols [search index=_internal sourcetype=scheduler thread_id=AlertNotifier* savedsearch_name="PUTYOURALERTSEARCHNAMEHERE" earliest=@d | head 1 | table _time]  | where '% used' > 75 AND isnull(_time)| fields "% used", "updated" 

The appendcols subsearch will try to find the timestamp of an alert that was fired today. If it finds one, the alert will be be fired.

splunkfmpa
New Member

Thank you, I inserted the search and will test today. I sent ahead and removed any throttle in the Trigger Condition screen.

0 Karma

fmpa_isaac
Path Finder

It was able to alert me based on the threshold I set. However, it continues to send an alert. How would you schedule this report in the Type and Trigger condition?

0 Karma

somesoni2
Revered Legend

What is your alert search ?

0 Karma

splunkfmpa
New Member
| rest /services/licenser/usage | eval "% used"=round(slaves_usage_bytes/quota*100,2) | where '% used' > 75 | fields "% used", "updated"
0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to November Tech Talks, Office Hours, and Webinars!

🍂 Fall into November with a fresh lineup of Community Office Hours, Tech Talks, and Webinars we’ve ...

Transform your security operations with Splunk Enterprise Security

Hi Splunk Community, Splunk Platform has set a great foundation for your security operations. With the ...

Splunk Admins and App Developers | Earn a $35 gift card!

Splunk, in collaboration with ESG (Enterprise Strategy Group) by TechTarget, is excited to announce a ...