Why is my real-time email alert on Response Times not being triggered?

[adubblenie]

Hello,

I'm trying to set up an email alert to trigger when my response time hits 10s or more 2 times in 15 minutes. To test the alert, I have made it more than .1s. I have not been able to get the alert to work for the past 2 days, even though I can complete the search and see events occurring .

My search is:

source="My_Source" earliest=-5m | stats avg(ResponseTime) as value | where value > .1

It is a real-time Alert with a custom trigger condition of search count>2 in 15 minutes. I need a throttle on it, so I have been playing around with those settings, but nothing has seemed to trigger it.

Any suggestions on what might be happening?

Thanks in advance.

[masonmorales]

If the alert was working >2 days ago, you might have an issue with search concurrency. Do you have a lot of other scheduled and/or real-time searches running? Every real-time search consumes a CPU core, and there is a concurrent search limit both at the user level, and at the search head (global) level. You can download SoS (https://splunkbase.splunk.com/app/748/) to troubleshoot search concurrency, or SUM (https://splunkbase.splunk.com/app/2678/) to troubleshoot scheduled searches not running. You need access to index=_internal to use either app.