Alerting

Alert throttling capabilities

JIrojas
Explorer

Hi,

Found an issue that I was not able to work around with the alert thottling.

Given a search that works like this:

| mstats avg(_value) as value WHERE metric_name="disk.used_percent"  AND span=5m by host, path
| eval "Disk Used (%)"=round(value,2)
| search "Disk Used (%)" >= 90 AND "Disk Used (%)" < 95
| table host, path, "Disk Used (%)"

If I set the throttling to "per result" the problem is if 50 hosts crossed the threshold, I would get 50 individual alerts, in the case I set it up with emails, it would be 50 emails, which in this particular case is non-desirable.

If I set the trigger condition to "Once" instead of "for each result", I would now get a single e-mail with 50 instances in the "in-line table" but  the problem I get is that some alerting may be missing during the throttling time, like new instances that reach the threshold during the throttling period (which is the use case we want to solve with the "per result" throttling).

Basically what I need is a solution that can give a "smart throttling", silencing alerts based on hosts that have already triggered the alert, and also have a way to clamp all of the occurences at a given point in a single alert event, if possible.

Thanks!

Labels (4)
0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer Certification at ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...