Alerting

Splunk alert trigger issue

Kareem_Naeem
Loves-to-Learn

my alert is not triggered even with many matching events here are the details:

 


Screenshot from 2024-10-14 21-21-27.pngScreenshot from 2024-10-14 21-28-28.png

while the activity that generate these logs is running the real time alert is processing and found those events in the screenshot. i have waited for 5 mins and same issue

i have also tries scheduled and still the same issue no triggering

Labels (1)
0 Karma

Kareem_Naeem
Loves-to-Learn

and here i have tried the scheduled alert and here is the logs related to that alert:

log 1:

10-14-2024 22:16:01.088 +0400 INFO SavedSplunker - AlertNotifier::notifySearchCompleted: called for sid=scheduler__kareem__search__kareem_at_1728929760_35, condition=1


log 2:


10-14-2024 22:16:01.809 +0400 INFO SavedSplunker - savedsearch_id="nobody;search;kareem", search_type="scheduled", search_streaming=0, user="kareem", app="search", savedsearch_name="kareem", priority=default, status=success, digest_mode=1, durable_cursor=0, scheduled_time=1728929760, window_time=0, dispatch_time=1728929760, run_time=0.178, result_count=1531, alert_actions="email", sid="scheduler__kareem__search__kareem_at_1728929760_35", suppressed=0, action_time_ms=718, thread_id="AlertNotifierWorker-0", workload_pool=""

0 Karma

PickleRick
SplunkTrust
SplunkTrust

1. Please, don't use real-time alerts. Except for very very rare cases real-time searches should be avoided altogether.

2. How did you confirm that the alerts weren't triggered? Is the email the only action you have defined? Maybe the problem is with your action, not the alert itself.

0 Karma

Kareem_Naeem
Loves-to-Learn

email was the only action 

and how did i know is by checking the triggered alerts secion under activity, and as i said i tried scueduled alerts the same issue 

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

You must explicitly add an action to add an alert to triggered alerts. So if your only action was email, you must check why the email wasn't delivered (look in _internal for sendemail.py). At first glance your logs suggest that the alert notifier was actually dispatched.

0 Karma

Kareem_Naeem
Loves-to-Learn

i have had an error in the email logs and i have fized it and now im receiving emails correctly in my local mail server but still the alert when triggered it's not shown in the triggered alerts 

0 Karma
Get Updates on the Splunk Community!

Building Reliable Asset and Identity Frameworks in Splunk ES

 Accurate asset and identity resolution is the backbone of security operations. Without it, alerts are ...

Cloud Monitoring Console - Unlocking Greater Visibility in SVC Usage Reporting

For Splunk Cloud customers, understanding and optimizing Splunk Virtual Compute (SVC) usage and resource ...

Automatic Discovery Part 3: Practical Use Cases

If you’ve enabled Automatic Discovery in your install of the Splunk Distribution of the OpenTelemetry ...