Alerting

How to set up an alert if the same error occurs more than 2 times in one hour per server?

dolejh76
Communicator

I saw that someone asked something similar before but it was in reference to different data and I couldn't get it to work. Right now I am pulling all the data from windows servers. I would like to set up an alert if the same error (or warning) occurs more than twice in one hour per server. obvious this would not be for informational items etc.

This would run every hour

So for example

Host 1 - error1234@13:04 error1234@13:10 error1234@13:14
Host 2 - error1234@13:04

Alert sent at 14:00 - Host 1 has had error1234 occur x times from 13:00 to 14:00

Appreciate the help - thanks!

Tags (2)
0 Karma
1 Solution

MuS
Legend

Hi dolejh76,

Try something like this:

Your base search here AND error1234 | bucket _time span=1h | stats count by host | where count>2

Hope this helps...

cheers, MuS

View solution in original post

MuS
Legend

Hi dolejh76,

Try something like this:

Your base search here AND error1234 | bucket _time span=1h | stats count by host | where count>2

Hope this helps...

cheers, MuS

dolejh76
Communicator

Ok I manage to make forward progress...

index="wineventlog" AND Type=Error OR Type=Warning| bucket _time span=1h | stats count by host | where count>2

This results in a list of hosts and the count of messages over 2 times per hour.

How would I go about getting the output to be something like...

Host1
Error1234 - 10x
Error2345 - 7x
Warning1234 - 10x

Host1
Error1234 - 3x
Error2345 - 7x
Warning1234 - 6x

etc

The end goal will be to send these alerts to the helpdesk so that they see that there are continuous errors that are occurring frequently or warnings on a server so that we look at the root cause and resolve.

Thanks
John

0 Karma

dolejh76
Communicator

index="wineventlog" AND Type=Error OR Type=Warning | bucket _time span=1h | stats count by host,Message | where count>2

Think this gives me what I want. At least gives me a lot to look at...

John

0 Karma

MuS
Legend

Glad to see you got what you want 🙂

0 Karma

dolejh76
Communicator

How do I specify type error and type warning?

0 Karma

dolejh76
Communicator

I guess I should clarify - any error, not just 1234... - so do I just remove "AND error1234" from your string?

Thanks

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...