Alerting

Email alert not triggering

twiggle
Explorer

Hi I need help with my email alerts.

I basically need to have an email alerting me that one of my process which I am logging is taking more than 2 hours or x hours.

So I have the basic query set up and let's say it is QUERY.

I've made the following alert from the following query:

QUERY | eval result=if(x>2,"YES","NO") | table result
where x is the current time since the process started (in hours).

I then saved this query as an alert and used the following settings:
Alert type: real time
Trigger condition: custom
Custom condition: search result=YES
in: 2 day(s)

I verified that the search query:

QUERY | eval result=if(x>2,"YES","NO") | table result | search result=YES

gives me a result if the time taken is more than 2 hours however it doesn't trigger an email alert.

Anyone can give me an idea of what I did wrong or where I can go from here?

Tags (2)
0 Karma
1 Solution

woodcock
Esteemed Legend

First of all, unless you deliberately engineered your Splunk cluster for RealTime searches, DO NOT USE THEM; you will destroy performance on your entire cluster. I would run your search instead every 5 or 10 minutes over the last X hours. Next, I would use "QUERY | where x>2" and then use trigger conditions for "number of results > 0".

View solution in original post

woodcock
Esteemed Legend

First of all, unless you deliberately engineered your Splunk cluster for RealTime searches, DO NOT USE THEM; you will destroy performance on your entire cluster. I would run your search instead every 5 or 10 minutes over the last X hours. Next, I would use "QUERY | where x>2" and then use trigger conditions for "number of results > 0".

twiggle
Explorer

How do you get it to search every 5 or 10 minutes?

I looked at the schedule alert type and under the 'Time Range' there's only 'Run every hour', '.. Day', '... Week' etc.

0 Karma

twiggle
Explorer

Ah ok, using the cron notation for scheduled alerts right?

*/5 * * * * or */10 * * * *

0 Karma

woodcock
Esteemed Legend

Yes, "/5" works.

0 Karma

jeremiahc4
Builder

Are you verifying the search in real-time the same way you are scheduling it? I wonder if real-time can't keep track that far out.

0 Karma

twiggle
Explorer

Yes that's what I did to verify it. I did it that way as I read that the custom condition applies the query that you insert above the base query.

Which in this case is: QUERY | eval result=if(x>2,"YES","NO") | table result

I did ensure that the real-time search looks at records beyond 2 hours.

I'll look into what @woodcock mentioned. That seems to be a better alternative.

0 Karma

MichaelPriest
Communicator

Try with == instead of =, I'm not sure if this will help?

0 Karma

twiggle
Explorer

Nope, that didn't do the trick.

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...