Alerting

Splunk alert trigger issue

Kareem_Naeem
Loves-to-Learn

my alert is not triggered even with many matching events here are the details:

 


Screenshot from 2024-10-14 21-21-27.pngScreenshot from 2024-10-14 21-28-28.png

while the activity that generate these logs is running the real time alert is processing and found those events in the screenshot. i have waited for 5 mins and same issue

i have also tries scheduled and still the same issue no triggering

Labels (1)
0 Karma

Kareem_Naeem
Loves-to-Learn

and here i have tried the scheduled alert and here is the logs related to that alert:

log 1:

10-14-2024 22:16:01.088 +0400 INFO SavedSplunker - AlertNotifier::notifySearchCompleted: called for sid=scheduler__kareem__search__kareem_at_1728929760_35, condition=1


log 2:


10-14-2024 22:16:01.809 +0400 INFO SavedSplunker - savedsearch_id="nobody;search;kareem", search_type="scheduled", search_streaming=0, user="kareem", app="search", savedsearch_name="kareem", priority=default, status=success, digest_mode=1, durable_cursor=0, scheduled_time=1728929760, window_time=0, dispatch_time=1728929760, run_time=0.178, result_count=1531, alert_actions="email", sid="scheduler__kareem__search__kareem_at_1728929760_35", suppressed=0, action_time_ms=718, thread_id="AlertNotifierWorker-0", workload_pool=""

0 Karma

PickleRick
SplunkTrust
SplunkTrust

1. Please, don't use real-time alerts. Except for very very rare cases real-time searches should be avoided altogether.

2. How did you confirm that the alerts weren't triggered? Is the email the only action you have defined? Maybe the problem is with your action, not the alert itself.

0 Karma

Kareem_Naeem
Loves-to-Learn

email was the only action 

and how did i know is by checking the triggered alerts secion under activity, and as i said i tried scueduled alerts the same issue 

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

You must explicitly add an action to add an alert to triggered alerts. So if your only action was email, you must check why the email wasn't delivered (look in _internal for sendemail.py). At first glance your logs suggest that the alert notifier was actually dispatched.

0 Karma

Kareem_Naeem
Loves-to-Learn

i have had an error in the email logs and i have fized it and now im receiving emails correctly in my local mail server but still the alert when triggered it's not shown in the triggered alerts 

0 Karma
Get Updates on the Splunk Community!

Unlock Database Monitoring with Splunk Observability Cloud

  In today’s fast-paced digital landscape, even minor database slowdowns can disrupt user experiences and ...

Purpose in Action: How Splunk Is Helping Power an Inclusive Future for All

At Cisco, purpose isn’t a tagline—it’s a commitment. Cisco’s FY25 Purpose Report outlines how the company is ...

[Upcoming Webinar] Demo Day: Transforming IT Operations with Splunk

Join us for a live Demo Day at the Cisco Store on January 21st 10:00am - 11:00am PST In the fast-paced world ...