topic Re: Alert not triggering, please help! in Alerting

Alert not triggering, please help!

imggnz — Wed, 30 Jun 2021 02:56:27 GMT

I have the below alert

| tstats latest(_time) as latest where index=*rsa* earliest=-10m by index
| eval recent = if(latest > relative_time(now(),"-10m"),1,0), realLatest = strftime(latest,"%c")
| where recent = 0

triggering on a cron job (*/10 * * * *), set to alert when results are not equal to 0

I can force the query to return a result by modifying it to the below

| tstats latest(_time) as latest where index=*rsa* earliest=-0m by index
| eval recent = if(latest > relative_time(now(),"-0m"),1,0), realLatest = strftime(latest,"%c")
| where recent = 0

In both cases where the original query/alert returns a result (under the statistics tab), and the modified/forced query/alert does, a triggered alert does not seem to proc as well as the email and pagerduty notification actions also tied into the alert actions.

As far as I can tell this makes logical sense to me, can anybody please advise?

Re: Alert not triggering, please help!

kamlesh_vaghela — Thu, 01 Jul 2021 07:55:34 GMT

@imggnz

Can you please check logs using below search?

index=_internal sourcetype=scheduler savedsearch_name=YOUR_ALERT_NAME

Re: Alert not triggering, please help!

imggnz — Sun, 04 Jul 2021 22:52:38 GMT

07-05-2021 10:20:10.544 +1200 INFO SavedSplunker - savedsearch_id="nobody;WestpacAlerts;WNZL_PROTECT_RSA_NO_LOGS", search_type="scheduled", user="m773827", app="WestpacAlerts", savedsearch_name="WNZL_PROTECT_RSA_NO_LOGS", priority=default, status=success, digest_mode=1, scheduled_time=1625437200, window_time=0, dispatch_time=1625437209, run_time=0.416, result_count=0, alert_actions="", sid="scheduler__m773827__WestpacAlerts__RMD5c7bcdf14062491da_at_1625437200_73932", suppressed=0, thread_id="AlertNotifierWorker-0", workload_pool=""

^ When not triggered

07-05-2021 10:40:10.097 +1200 INFO SavedSplunker - savedsearch_id="nobody;WestpacAlerts;WNZL_PROTECT_RSA_NO_LOGS", search_type="scheduled", user="m773827", app="WestpacAlerts", savedsearch_name="WNZL_PROTECT_RSA_NO_LOGS", priority=default, status=success, digest_mode=1, scheduled_time=1625438400, window_time=0, dispatch_time=1625438407, run_time=0.418, result_count=1, alert_actions="email,pagerduty", sid="scheduler__m773827__WestpacAlerts__RMD5c7bcdf14062491da_at_1625438400_74194", suppressed=0, thread_id="AlertNotifierWorker-1", workload_pool=""

^ When triggered

I have resolved this issue by changing the trigger action, previously it was set to trigger when results were greater than zero, I changed to not equal to zero and now it works. Odd considering both sets of logic would work in my mind.

Thank you for your help KV!