Alerting

Alert not triggering, please help!

imggnz
Engager

I have the below alert

| tstats latest(_time) as latest where index=*rsa* earliest=-10m by index
| eval recent = if(latest > relative_time(now(),"-10m"),1,0), realLatest = strftime(latest,"%c")
| where recent = 0

triggering on a cron job (*/10 * * * *), set to alert when results are not equal to 0

I can force the query to return a result by modifying it to the below

| tstats latest(_time) as latest where index=*rsa* earliest=-0m by index
| eval recent = if(latest > relative_time(now(),"-0m"),1,0), realLatest = strftime(latest,"%c")
| where recent = 0

In both cases where the original query/alert returns a result (under the statistics tab), and the modified/forced query/alert does, a triggered alert does not seem to proc as well as the email and pagerduty notification actions also tied into the alert actions.

As far as I can tell this makes logical sense to me, can anybody please advise?

Labels (2)
0 Karma
1 Solution

imggnz
Engager

07-05-2021 10:20:10.544 +1200 INFO SavedSplunker - savedsearch_id="nobody;WestpacAlerts;WNZL_PROTECT_RSA_NO_LOGS", search_type="scheduled", user="m773827", app="WestpacAlerts", savedsearch_name="WNZL_PROTECT_RSA_NO_LOGS", priority=default, status=success, digest_mode=1, scheduled_time=1625437200, window_time=0, dispatch_time=1625437209, run_time=0.416, result_count=0, alert_actions="", sid="scheduler__m773827__WestpacAlerts__RMD5c7bcdf14062491da_at_1625437200_73932", suppressed=0, thread_id="AlertNotifierWorker-0", workload_pool=""

^ When not triggered

07-05-2021 10:40:10.097 +1200 INFO SavedSplunker - savedsearch_id="nobody;WestpacAlerts;WNZL_PROTECT_RSA_NO_LOGS", search_type="scheduled", user="m773827", app="WestpacAlerts", savedsearch_name="WNZL_PROTECT_RSA_NO_LOGS", priority=default, status=success, digest_mode=1, scheduled_time=1625438400, window_time=0, dispatch_time=1625438407, run_time=0.418, result_count=1, alert_actions="email,pagerduty", sid="scheduler__m773827__WestpacAlerts__RMD5c7bcdf14062491da_at_1625438400_74194", suppressed=0, thread_id="AlertNotifierWorker-1", workload_pool=""

^ When triggered

I have resolved this issue by changing the trigger action, previously it was set to trigger when results were greater than zero, I changed to not equal to zero and now it works. Odd considering both sets of logic would work in my mind.

Thank you for your help KV!

View solution in original post

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

@imggnz 

Can you please check logs using below search?

index=_internal sourcetype=scheduler savedsearch_name=YOUR_ALERT_NAME

 

KV

0 Karma

imggnz
Engager

07-05-2021 10:20:10.544 +1200 INFO SavedSplunker - savedsearch_id="nobody;WestpacAlerts;WNZL_PROTECT_RSA_NO_LOGS", search_type="scheduled", user="m773827", app="WestpacAlerts", savedsearch_name="WNZL_PROTECT_RSA_NO_LOGS", priority=default, status=success, digest_mode=1, scheduled_time=1625437200, window_time=0, dispatch_time=1625437209, run_time=0.416, result_count=0, alert_actions="", sid="scheduler__m773827__WestpacAlerts__RMD5c7bcdf14062491da_at_1625437200_73932", suppressed=0, thread_id="AlertNotifierWorker-0", workload_pool=""

^ When not triggered

07-05-2021 10:40:10.097 +1200 INFO SavedSplunker - savedsearch_id="nobody;WestpacAlerts;WNZL_PROTECT_RSA_NO_LOGS", search_type="scheduled", user="m773827", app="WestpacAlerts", savedsearch_name="WNZL_PROTECT_RSA_NO_LOGS", priority=default, status=success, digest_mode=1, scheduled_time=1625438400, window_time=0, dispatch_time=1625438407, run_time=0.418, result_count=1, alert_actions="email,pagerduty", sid="scheduler__m773827__WestpacAlerts__RMD5c7bcdf14062491da_at_1625438400_74194", suppressed=0, thread_id="AlertNotifierWorker-1", workload_pool=""

^ When triggered

I have resolved this issue by changing the trigger action, previously it was set to trigger when results were greater than zero, I changed to not equal to zero and now it works. Odd considering both sets of logic would work in my mind.

Thank you for your help KV!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...