Alerting

Why does my real-time alert continue to send emails for events that happened hours ago?

di2esysadmin
Path Finder

I have a simple search:

host=*prod*  "Too many open files"  source!="/opt/atlassian/jira-data/log/emh.log*"

I've set up a simple alert. Real-time, throttling on host for 1 minute. (pic attached)

We had 200+ of these errors, all within 2 seconds of each other, from a single host 2 hours ago. Since then, we've received no less than 24 emails alerting us of those events. I want one email, not 24.

What have I misconfigured?

Thanks!

alt text

0 Karma
1 Solution

linu1988
Champion

Hi,
Please try to set the below setting with your preferred field. Always try to table to data you require. Moreover the throttling period is only 1 minute, so if you have events matching it will generate the alert. If you want hourly then throttle it to send 1 hour per host in the suppress triggering option.

alt text

Thanks,
L

View solution in original post

0 Karma

linu1988
Champion

Hi,
Please try to set the below setting with your preferred field. Always try to table to data you require. Moreover the throttling period is only 1 minute, so if you have events matching it will generate the alert. If you want hourly then throttle it to send 1 hour per host in the suppress triggering option.

alt text

Thanks,
L

0 Karma

di2esysadmin
Path Finder

Thank you. I've made these changes. I've my fingers crossed this will do the trick.

0 Karma
Get Updates on the Splunk Community!

Observability Unveiled: Navigating OpenTelemetry's Framework and Deployment Options

Observability Unveiled: Navigating OpenTelemetry's Framework and Deployment Options A recent Tech Talk, ...

Observability | How to Think About Instrumentation Overhead (White Paper)

Novice observability practitioners are often overly obsessed with performance. They might approach ...

Cloud Platform | Get Resiliency in the Cloud Event (Register Now!)

IDC Report: Enterprises Gain Higher Efficiency and Resiliency With Migration to Cloud  Today many enterprises ...