Alerting

How to set up an alert if the same error occurs more than 2 times in one hour per server?

dolejh76
Communicator

I saw that someone asked something similar before but it was in reference to different data and I couldn't get it to work. Right now I am pulling all the data from windows servers. I would like to set up an alert if the same error (or warning) occurs more than twice in one hour per server. obvious this would not be for informational items etc.

This would run every hour

So for example

Host 1 - error1234@13:04 error1234@13:10 error1234@13:14
Host 2 - error1234@13:04

Alert sent at 14:00 - Host 1 has had error1234 occur x times from 13:00 to 14:00

Appreciate the help - thanks!

Tags (2)
0 Karma
1 Solution

MuS
Legend

Hi dolejh76,

Try something like this:

Your base search here AND error1234 | bucket _time span=1h | stats count by host | where count>2

Hope this helps...

cheers, MuS

View solution in original post

MuS
Legend

Hi dolejh76,

Try something like this:

Your base search here AND error1234 | bucket _time span=1h | stats count by host | where count>2

Hope this helps...

cheers, MuS

dolejh76
Communicator

Ok I manage to make forward progress...

index="wineventlog" AND Type=Error OR Type=Warning| bucket _time span=1h | stats count by host | where count>2

This results in a list of hosts and the count of messages over 2 times per hour.

How would I go about getting the output to be something like...

Host1
Error1234 - 10x
Error2345 - 7x
Warning1234 - 10x

Host1
Error1234 - 3x
Error2345 - 7x
Warning1234 - 6x

etc

The end goal will be to send these alerts to the helpdesk so that they see that there are continuous errors that are occurring frequently or warnings on a server so that we look at the root cause and resolve.

Thanks
John

0 Karma

dolejh76
Communicator

index="wineventlog" AND Type=Error OR Type=Warning | bucket _time span=1h | stats count by host,Message | where count>2

Think this gives me what I want. At least gives me a lot to look at...

John

0 Karma

MuS
Legend

Glad to see you got what you want 🙂

0 Karma

dolejh76
Communicator

How do I specify type error and type warning?

0 Karma

dolejh76
Communicator

I guess I should clarify - any error, not just 1234... - so do I just remove "AND error1234" from your string?

Thanks

0 Karma
Get Updates on the Splunk Community!

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...

Adoption of Infrastructure Monitoring at Splunk

  Splunk's Growth Engineering team showcases one of their first Splunk product adoption-Splunk Infrastructure ...