Alerting

Why does my real-time alert continue to send emails for events that happened hours ago?

di2esysadmin
Path Finder

I have a simple search:

host=*prod*  "Too many open files"  source!="/opt/atlassian/jira-data/log/emh.log*"

I've set up a simple alert. Real-time, throttling on host for 1 minute. (pic attached)

We had 200+ of these errors, all within 2 seconds of each other, from a single host 2 hours ago. Since then, we've received no less than 24 emails alerting us of those events. I want one email, not 24.

What have I misconfigured?

Thanks!

alt text

0 Karma
1 Solution

linu1988
Champion

Hi,
Please try to set the below setting with your preferred field. Always try to table to data you require. Moreover the throttling period is only 1 minute, so if you have events matching it will generate the alert. If you want hourly then throttle it to send 1 hour per host in the suppress triggering option.

alt text

Thanks,
L

View solution in original post

0 Karma

linu1988
Champion

Hi,
Please try to set the below setting with your preferred field. Always try to table to data you require. Moreover the throttling period is only 1 minute, so if you have events matching it will generate the alert. If you want hourly then throttle it to send 1 hour per host in the suppress triggering option.

alt text

Thanks,
L

0 Karma

di2esysadmin
Path Finder

Thank you. I've made these changes. I've my fingers crossed this will do the trick.

0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...