I am currently sending all cisco ace load balancer syslogs to my splunk server.
Within Splunk, I have two separate real-time alerts - one alert notifies me via email when a certain server goes down and a separate alert notifies me when the server comes back up.
Is it possible to create a custom alert where I will only be notified if the server does not come back up after being down for more than X amount of hours? Receiving up down alerts is very annoying and sometimes there are so many emails, I wouldn't know if an up alert matches a down alert.
If this is possible, how would I go about implementing it? Thanks
To provide a little more detail, here is exactly what my real-time alerts look like:
Alert 1 - "Particular Server Name" Changed State to DOWN - send email
Alert 2 - "Particular Server Name" Changed State to UP - send email
Where the server name is an arbitrary name of a server that wouldn't mean anything to anybody
even if I did copy it directly from my alert.
Sometimes the patching team fails to bring up a server properly and we find out the hard way when somebody complains. I actually have dozens of alerts just like this but for different servers. However, one solution would apply for all of my alerts.
... View more