Alerting

Alert Trigger Question | Don't trigger it for that specific user for x amount of minutes

ycefalas
Loves-to-Learn Lots

I made an alert query that particularly looks for a windows failed login by users using stats. It works.

Whenever there is an event greater than 0, it’ll show case it and display it. It works.

Now here comes the problem:

The user who is constantly failing over a period of time also causes mass amount of alert notification triggers. Let’s say it’s every 10 minutes for the alert interval. Every 10 minutes we’ll be notified for the same user failing.

There is this option in splunk, that I am aware of:
alt text

This option works per say, however, if now a different user account were to have a +1 count, it will not be alerted because the alert won’t trigger until the next 20 minutes.

So here comes the question:

How can I make alert triggers intelligent enough to distinguish each user account as unique but if the user account was last seen then don’t trigger that same account for X amount of hours?

Hopefully I made sense, if not I’ll try to elaborate the problem further:

Account1 failed 5 logins at 1:00 triggered
Account2 failed 10 logins at 1:10 no trigger because of “after triggering the alert, don’t trigger it again for 20 minutes…”
Account1 failed 5 logins at 1:20 triggered

Tags (2)
0 Karma

DalJeanis
Legend

If you have a user who is "constantly failing" over a period of time, then that is a training problem.

If your job is running every 10 minutes, across a 10-minute timeframe, and deciding whether to alert, then you can just change it to run across a 20-minute timeframe, and alert only for those users that deserve an alert in the second 10 minutes but did not deserve (and therefore probably receive) an alert in the first 10 minutes. It won't be the splunk alert that's suppressing the long-term fails, but the search itself.across

You could actually go one further, just in case someone keeps failing long-term. Do the calculation across a 30 minute period. If the present period is an alert, suppress the alert only if the prior period was an alert but two periods ago was NOT an alert. Basically, if the guy has been failing for 30 minutes straight then there is something really wrong with him and we should send Mongo to go break his legs...

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Observability Simplified: Combining User Experience, Application Performance & ...

Tech Talk Observability Simplified: Combining User Experience, Application Performance & Network ...

Event Series May & June: From Network Visibility to Service Intelligence

Unifying the Network: Moving from Alert Noise to Service Intelligence with Splunk ITSI In today’s hybrid ...

Global Splunk User Group Events: May + June 2026

Your Splunk Community Awaits: Discover Upcoming User Group Events Worldwide    Staying ahead in the fast-paced ...