topic Re: Email alert not triggering in Alerting

Email alert not triggering

twiggle — Tue, 05 May 2015 11:59:21 GMT

Hi I need help with my email alerts.

I basically need to have an email alerting me that one of my process which I am logging is taking more than 2 hours or x hours.

So I have the basic query set up and let's say it is QUERY.

I've made the following alert from the following query:

QUERY | eval result=if(x>2,"YES","NO") | table result
where x is the current time since the process started (in hours).

I then saved this query as an alert and used the following settings:
Alert type: real time
Trigger condition: custom
Custom condition: search result=YES
in: 2 day(s)

I verified that the search query:

QUERY | eval result=if(x>2,"YES","NO") | table result | search result=YES

gives me a result if the time taken is more than 2 hours however it doesn't trigger an email alert.

Anyone can give me an idea of what I did wrong or where I can go from here?

Re: Email alert not triggering

MichaelPriest — Tue, 05 May 2015 13:11:17 GMT

Try with == instead of =, I'm not sure if this will help?

Re: Email alert not triggering

jeremiahc4 — Tue, 05 May 2015 14:21:46 GMT

Are you verifying the search in real-time the same way you are scheduling it? I wonder if real-time can't keep track that far out.

Re: Email alert not triggering

woodcock — Tue, 05 May 2015 15:31:27 GMT

First of all, unless you deliberately engineered your Splunk cluster for RealTime searches, DO NOT USE THEM; you will destroy performance on your entire cluster. I would run your search instead every 5 or 10 minutes over the last X hours. Next, I would use "QUERY | where x>2" and then use trigger conditions for "number of results > 0".

Re: Email alert not triggering

twiggle — Wed, 06 May 2015 08:13:10 GMT

How do you get it to search every 5 or 10 minutes?

I looked at the schedule alert type and under the 'Time Range' there's only 'Run every hour', '.. Day', '... Week' etc.

Re: Email alert not triggering

twiggle — Wed, 06 May 2015 08:17:54 GMT

Yes that's what I did to verify it. I did it that way as I read that the custom condition applies the query that you insert above the base query.

Which in this case is: QUERY | eval result=if(x>2,"YES","NO") | table result

I did ensure that the real-time search looks at records beyond 2 hours.

I'll look into what @woodcock mentioned. That seems to be a better alternative.

Re: Email alert not triggering

twiggle — Wed, 06 May 2015 08:48:18 GMT

Nope, that didn't do the trick.

Re: Email alert not triggering

twiggle — Wed, 06 May 2015 08:49:52 GMT

Ah ok, using the cron notation for scheduled alerts right?

*/5 * * * * or */10 * * * *

Re: Email alert not triggering

woodcock — Wed, 06 May 2015 15:23:45 GMT

Yes, "/5" works.