Hi I need help with my email alerts.
I basically need to have an email alerting me that one of my process which I am logging is taking more than 2 hours or x hours.
So I have the basic query set up and let's say it is QUERY.
I've made the following alert from the following query:
QUERY | eval result=if(x>2,"YES","NO") | table result
where x is the current time since the process started (in hours).
I then saved this query as an alert and used the following settings:
Alert type: real time
Trigger condition: custom
Custom condition: search result=YES
in: 2 day(s)
I verified that the search query:
QUERY | eval result=if(x>2,"YES","NO") | table result | search result=YES
gives me a result if the time taken is more than 2 hours however it doesn't trigger an email alert.
Anyone can give me an idea of what I did wrong or where I can go from here?
First of all, unless you deliberately engineered your Splunk cluster for RealTime searches, DO NOT USE THEM; you will destroy performance on your entire cluster. I would run your search instead every 5 or 10 minutes over the last X hours. Next, I would use "QUERY | where x>2" and then use trigger conditions for "number of results > 0".
First of all, unless you deliberately engineered your Splunk cluster for RealTime searches, DO NOT USE THEM; you will destroy performance on your entire cluster. I would run your search instead every 5 or 10 minutes over the last X hours. Next, I would use "QUERY | where x>2" and then use trigger conditions for "number of results > 0".
How do you get it to search every 5 or 10 minutes?
I looked at the schedule alert type and under the 'Time Range' there's only 'Run every hour', '.. Day', '... Week' etc.
Ah ok, using the cron notation for scheduled alerts right?
*/5 * * * * or */10 * * * *
Yes, "/5" works.
Are you verifying the search in real-time the same way you are scheduling it? I wonder if real-time can't keep track that far out.
Yes that's what I did to verify it. I did it that way as I read that the custom condition applies the query that you insert above the base query.
Which in this case is: QUERY | eval result=if(x>2,"YES","NO") | table result
I did ensure that the real-time search looks at records beyond 2 hours.
I'll look into what @woodcock mentioned. That seems to be a better alternative.
Try with == instead of =, I'm not sure if this will help?
Nope, that didn't do the trick.