Alerting

How to not create an alert for repeated events over a day

Toshbar
Explorer

Im currently running an alert, which updates every minute with a range -1m to -2m, for each new log based on unique JOBNAMEs. I want to create an alert each time a new JOBNAME occurs for the first time but not again if the same JOBNAME occurs for a given day.
Dedup is used to remove duplicates and I can use it in order to only have unique JOBNAMES per 24 hour period but this won't only show the first time entry per JOBNAME. I'm thinking I need to do my regular search, with the alert parameters set to every minute, span -1m to -2m, for each result and then NOT that against the results found in the same day based on JOBNAME. Any help/ideas?

Here is the search result for a given day:

index = x 
[ | inputlookup Jobnames.csv | fields JOBNAME ] 

JOBNAME DATETIME
CIPB0021    2017-07-31 20:41:07.20 -0700
CIPB0024    2017-07-31 20:45:59.69 -0700
CIPB0021    2017-07-31 20:48:15.50 -0700
CIPB0024    2017-07-31 20:54:42.04 -0700
CIPB0024    2017-07-31 20:57:25.70 -0700
CIPB0021    2017-07-31 20:58:25.80 -0700
CIPB0021    2017-07-31 21:02:59.15 -0700
CIPB0024    2017-07-31 21:03:28.13 -0700
CIBI0991    2017-07-31 21:16:59.60 -0700
CIBI0991    2017-07-31 21:40:29.76 -0700

Here is the search result using dedup:

index = x 
[ | inputlookup Jobnames.csv | fields JOBNAME ] 
| bucket _time span=24hr 
| dedup JOBNAME _time

JOBNAME DATETIME
CIPB0021    2017-07-31 20:58:25.80 -0700
CIPB0024    2017-07-31 21:03:28.13 -0700
CIBI0991    2017-07-31 21:16:59.60 -0700

As you can see, CIPB0021 occurred 4 times, and it filtered to show the 3rd. CIPB0024 occurred 4 times, and it filtered to show the 4th. CIBI0991 occurred 2 times and it showed the 1st. Also bucket uses the last 24 hours and not only the same day. So if something occurs at 1am, it would use most of the previous day in the comparison which I don't want

I only want to create alerts for the following:

JOBNAME DATETIME
CIPB0021    2017-07-31 20:41:07.20 -0700
CIPB0024    2017-07-31 20:45:59.69 -0700
CIBI0991    2017-07-31 21:16:59.60 -0700
0 Karma
1 Solution

hardikJsheth
Motivator

Use throttling mechanism and suppress alert for specific period of time.

View solution in original post

0 Karma

hardikJsheth
Motivator

Use throttling mechanism and suppress alert for specific period of time.

0 Karma

Toshbar
Explorer

If I throttle for the alert, how would I prevent it from running for rest of the day? The options are seconds/minutes/hours/days

0 Karma

hardikJsheth
Motivator

you need to set it to not generate an alert on JOBNAME for next 24 hours.

Toshbar
Explorer

Would that run over into the following day? Ex: alert triggers at 11pm -> set to ignore for 24 hours -> won't it ignore all of the following day up until 11pm? I want it to only ignore 1hr in that case or if an alert is triggered at 7pm ignore for 5 hrs.

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...