Alerting

Prevent repeat alerts

Branden
Builder

Hi. We have script that Splunk runs every 15 minutes. The script checks to see if a partition is using the primary or back-up ethernet adapter. If it is using the back-up adapter, it notifies us via e-mail.

This alert works great except for one thing: it will repeat the alert every time the script runs. That means we get alerted every 15 minutes, which creates a cluttered mailbox in the morning.

I realize that this may be expected behavior. I'm wondering if anyone has a procedure or trick to prevent a repeat alert from going out.

I'm running Splunk 4.1.5 on the indexer and 4.1.4 on the forwarders.

Thanks!

Tags (2)
1 Solution

ftk
Motivator

Take a look at the AlertThrottle app on Splunkbase, with it you will be able to throttle repeat alerts.

View solution in original post

lmalhoit
Explorer

How did you set it up so that it didn't notify you anymore? I've only been able to throttle it, not stop the notification all together.

0 Karma

ftk
Motivator

Take a look at the AlertThrottle app on Splunkbase, with it you will be able to throttle repeat alerts.

Branden
Builder

🙂
Actually, I went ahead and tried it anyways. I must admit it worked like a charm. Thank you for the tip!

0 Karma

ftk
Motivator

@Branden, the AlertThrottle app was written and is maintained by a Splunk employee. Not sure if that alleviates any reservations, but maybe it will 🙂

0 Karma

Branden
Builder

Thank you for the response. I'm a bit wary about relying on a 3rd party app for our production system (it is poorly documented, has no ratings, and I'm not sure how reliable it is).
Nevertheless, I am willing to give it a try. Unfortunately, I am not sure I can create the custom condition I'm looking for in the custom condition field. I'm sure it can be done, I'll just have to ask it in a separate question.
Thank you again.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...