topic Re: Prevent repeat alerts in Alerting

Prevent repeat alerts

Branden — Mon, 13 Sep 2010 19:37:43 GMT

Hi. We have script that Splunk runs every 15 minutes. The script checks to see if a partition is using the primary or back-up ethernet adapter. If it is using the back-up adapter, it notifies us via e-mail.

This alert works great except for one thing: it will repeat the alert every time the script runs. That means we get alerted every 15 minutes, which creates a cluttered mailbox in the morning.

I realize that this may be expected behavior. I'm wondering if anyone has a procedure or trick to prevent a repeat alert from going out.

I'm running Splunk 4.1.5 on the indexer and 4.1.4 on the forwarders.

Thanks!

Re: Prevent repeat alerts

ftk — Mon, 13 Sep 2010 20:07:12 GMT

Take a look at the AlertThrottle app on Splunkbase, with it you will be able to throttle repeat alerts.

Re: Prevent repeat alerts

Branden — Mon, 13 Sep 2010 20:34:21 GMT

Thank you for the response. I'm a bit wary about relying on a 3rd party app for our production system (it is poorly documented, has no ratings, and I'm not sure how reliable it is).
Nevertheless, I am willing to give it a try. Unfortunately, I am not sure I can create the custom condition I'm looking for in the custom condition field. I'm sure it can be done, I'll just have to ask it in a separate question.
Thank you again.

Re: Prevent repeat alerts

ftk — Mon, 13 Sep 2010 20:57:34 GMT

@Branden, the AlertThrottle app was written and is maintained by a Splunk employee. Not sure if that alleviates any reservations, but maybe it will 🙂

Re: Prevent repeat alerts

Branden — Mon, 13 Sep 2010 21:34:13 GMT

🙂
Actually, I went ahead and tried it anyways. I must admit it worked like a charm. Thank you for the tip!

Re: Prevent repeat alerts

lmalhoit — Thu, 14 Jul 2011 21:54:14 GMT

How did you set it up so that it didn't notify you anymore? I've only been able to throttle it, not stop the notification all together.