Splunk Enterprise

How to create Alert trigger for only when there is consistent failure rates

Path Finder

Hi, I have a requirement where I want to create an alert on some of my APIs which are being monitored in Splunk.

I've created a search which checks the success/failures of each API and then calculates the failure rate and if that is more than 10% then it triggers the alert.
Now what is happening is the alerts gets triggered even for bigger blips when they are only for short duration. Like there is a high increase in error rate for 5 mins and then it gets recovered itself. I don't want to trigger the alert in that situation because it will make unnecessary callouts to people for investigation which is not required.

How can i create alert which runs every 30 mins and looks into the failure rate consistently for each 5 mins in the last 30 minutes period. So if the failure rate is consistent for more than 15/20 mins then only trigger the alert.

This is my base search



index=api_prod (message.httpResponseCode=50* OR message.httpResponseCode=20*)
| rename message.serviceName as serviceName message.httpResponseCode as httpResponseCode 
| stats count as totalrequests count(eval(like(httpResponseCode, "20%"))) as successrequest count(eval(like(httpResponseCode, "50%"))) as failedrequest by serviceName 
| eval Total = successrequest + failedrequest 
| eval failureRatePercentage = round(((failedrequest/totalrequests) * 100),2) 
| where failureRatePercentage > 10
| fields - Total
|table serviceName,totalrequests,successrequest,failedrequest,failureRatePercentage



Any guidance is really appreciated.

Best Regards,

Labels (3)
0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...