Email alert not triggering

[twiggle]

Hi I need help with my email alerts.

I basically need to have an email alerting me that one of my process which I am logging is taking more than 2 hours or x hours.

So I have the basic query set up and let's say it is QUERY.

I've made the following alert from the following query:

QUERY | eval result=if(x>2,"YES","NO") | table result
where x is the current time since the process started (in hours).

I then saved this query as an alert and used the following settings:
Alert type: real time
Trigger condition: custom
Custom condition: search result=YES
in: 2 day(s)

I verified that the search query:

QUERY | eval result=if(x>2,"YES","NO") | table result | search result=YES

gives me a result if the time taken is more than 2 hours however it doesn't trigger an email alert.

Anyone can give me an idea of what I did wrong or where I can go from here?

[woodcock · Accepted solution]

First of all, unless you deliberately engineered your Splunk cluster for RealTime searches, DO NOT USE THEM; you will destroy performance on your entire cluster. I would run your search instead every 5 or 10 minutes over the last X hours. Next, I would use "QUERY | where x>2" and then use trigger conditions for "number of results > 0".

[twiggle]

How do you get it to search every 5 or 10 minutes?

I looked at the schedule alert type and under the 'Time Range' there's only 'Run every hour', '.. Day', '... Week' etc.

[twiggle]

Ah ok, using the cron notation for scheduled alerts right?

*/5 * * * * or */10 * * * *

[woodcock]

Yes, "/5" works.

[jeremiahc4]

Are you verifying the search in real-time the same way you are scheduling it? I wonder if real-time can't keep track that far out.

[twiggle]

Yes that's what I did to verify it. I did it that way as I read that the custom condition applies the query that you insert above the base query.

Which in this case is: QUERY | eval result=if(x>2,"YES","NO") | table result

I did ensure that the real-time search looks at records beyond 2 hours.

I'll look into what @woodcock mentioned. That seems to be a better alternative.

[MichaelPriest]

Try with == instead of =, I'm not sure if this will help?

[twiggle]

Nope, that didn't do the trick.