Splunk Enterprise

How to create Alert trigger for only when there is consistent failure rates

shashank_24
Path Finder

Hi, I have a requirement where I want to create an alert on some of my APIs which are being monitored in Splunk.

I've created a search which checks the success/failures of each API and then calculates the failure rate and if that is more than 10% then it triggers the alert.
Now what is happening is the alerts gets triggered even for bigger blips when they are only for short duration. Like there is a high increase in error rate for 5 mins and then it gets recovered itself. I don't want to trigger the alert in that situation because it will make unnecessary callouts to people for investigation which is not required.

How can i create alert which runs every 30 mins and looks into the failure rate consistently for each 5 mins in the last 30 minutes period. So if the failure rate is consistent for more than 15/20 mins then only trigger the alert.

This is my base search

 

 

index=api_prod (message.httpResponseCode=50* OR message.httpResponseCode=20*)
| rename message.serviceName as serviceName message.httpResponseCode as httpResponseCode 
| stats count as totalrequests count(eval(like(httpResponseCode, "20%"))) as successrequest count(eval(like(httpResponseCode, "50%"))) as failedrequest by serviceName 
| eval Total = successrequest + failedrequest 
| eval failureRatePercentage = round(((failedrequest/totalrequests) * 100),2) 
| where failureRatePercentage > 10
| fields - Total
|table serviceName,totalrequests,successrequest,failedrequest,failureRatePercentage

 

 

Any guidance is really appreciated.

Best Regards,
Shashank

Labels (2)
0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July Tech Talks, Office Hours, and Webinars!

What are Community Office Hours?Community Office Hours is an interactive 60-minute Zoom series where ...

Updated Data Type Articles, Anniversary Celebrations, and More on Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

A Prelude to .conf25: Your Guide to Splunk University

Heading to Boston this September for .conf25? Get a jumpstart by arriving a few days early for Splunk ...