Alerting

How do I suppress alerts until the next day at 12 am and not 24 hours?

splunkfmpa
New Member

I have a couple of alerts for License usage set to check every hour when they exceed 75 %. At the moment, I receive an email every hour after it reaches the threshold. I only want to be notified one time for that day. If I use the throttle option and suppress for 1 day, it will wait 24 hours until it alerts me again. This is an issue because if the threshold is exceeded earlier the following day, I won't get notified until that time runs out. I would like to suppress the alert until 12 am that night. Can anyone please help? Thank you

0 Karma

fredchase
Explorer

I posted a solution to this general problem here: https://answers.splunk.com/answers/337985/throttle-alert-once-per-day.html
In short, the search string I used to trigger once a day alerting when over the license limit is this:

| rest splunk_server=local /services/licenser/pools | rename title AS Pool | search [rest splunk_server=local /services/licenser/groups | search is_active=1 | eval stack_id=stack_ids | fields stack_id] | eval quota=if(isnull(effective_quota),quota,effective_quota) | eval "Percentage of daily license limit used"=round(used_bytes/quota*100,2) | eval "Alert time"=strftime(now(), "%T %Z") | eval alert_count_today=[search index=_internal sourcetype=scheduler thread_id=AlertNotifier* savedsearch_name="License Limit Exceeded: Over 100% Usage" earliest=@d | where alert_actions!="" | stats count | return($count)] | where 'Percentage of daily license limit used' > 100 and alert_count_today = 0 | fields "Alert time" "Percentage of daily license limit used"

For more details see the original post.

somesoni2
SplunkTrust
SplunkTrust

Try this as your alert search (no throttling needs to be set)

| rest /services/licenser/usage | eval "% used"=round(slaves_usage_bytes/quota*100,2) | appendcols [search index=_internal sourcetype=scheduler thread_id=AlertNotifier* savedsearch_name="PUTYOURALERTSEARCHNAMEHERE" earliest=@d | head 1 | table _time]  | where '% used' > 75 AND isnull(_time)| fields "% used", "updated" 

The appendcols subsearch will try to find the timestamp of an alert that was fired today. If it finds one, the alert will be be fired.

splunkfmpa
New Member

Thank you, I inserted the search and will test today. I sent ahead and removed any throttle in the Trigger Condition screen.

0 Karma

fmpa_isaac
Path Finder

It was able to alert me based on the threshold I set. However, it continues to send an alert. How would you schedule this report in the Type and Trigger condition?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

What is your alert search ?

0 Karma

splunkfmpa
New Member
| rest /services/licenser/usage | eval "% used"=round(slaves_usage_bytes/quota*100,2) | where '% used' > 75 | fields "% used", "updated"
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...