Alerting

If we have an alert that tells us when a service is stopped, how can we create an alert that will also check if the service remains down?

luise
New Member

Hello,

We have an alert that tells us when a service stopped. However, the services will sometimes restart. How can we have an alert that will also check if the service remains down?

Is it possible?

Thanks,
Luis Embalo

0 Karma

sundareshr
Legend

Maybe something like this might work

index=* "success" earliest=[search index=* "failed" | timechart latest(failed) as earliest | fields earliest] | stats count 

If the count is 0 its probably because the service hasn't restarted.

0 Karma

luise
New Member

That searches for a service that is down.. what I am looking for is for way that sees the service as going down, and then keeps watching to see if it came up. If it came up, then nothing happens.. if it did not come up, then alerts.

Is that possible?

0 Karma
Get Updates on the Splunk Community!

Monitoring Postgres with OpenTelemetry

Behind every business-critical application, you’ll find databases. These behind-the-scenes stores power ...

Mastering Synthetic Browser Testing: Pro Tips to Keep Your Web App Running Smoothly

To start, if you're new to synthetic monitoring, I recommend exploring this synthetic monitoring overview. In ...

Splunk Edge Processor | Popular Use Cases to Get Started with Edge Processor

Splunk Edge Processor offers more efficient, flexible data transformation – helping you reduce noise, control ...