Alerting

Splunk alert trigger issue

Kareem_Naeem
Loves-to-Learn

my alert is not triggered even with many matching events here are the details:

 


Screenshot from 2024-10-14 21-21-27.pngScreenshot from 2024-10-14 21-28-28.png

while the activity that generate these logs is running the real time alert is processing and found those events in the screenshot. i have waited for 5 mins and same issue

i have also tries scheduled and still the same issue no triggering

Labels (1)
0 Karma

Kareem_Naeem
Loves-to-Learn

and here i have tried the scheduled alert and here is the logs related to that alert:

log 1:

10-14-2024 22:16:01.088 +0400 INFO SavedSplunker - AlertNotifier::notifySearchCompleted: called for sid=scheduler__kareem__search__kareem_at_1728929760_35, condition=1


log 2:


10-14-2024 22:16:01.809 +0400 INFO SavedSplunker - savedsearch_id="nobody;search;kareem", search_type="scheduled", search_streaming=0, user="kareem", app="search", savedsearch_name="kareem", priority=default, status=success, digest_mode=1, durable_cursor=0, scheduled_time=1728929760, window_time=0, dispatch_time=1728929760, run_time=0.178, result_count=1531, alert_actions="email", sid="scheduler__kareem__search__kareem_at_1728929760_35", suppressed=0, action_time_ms=718, thread_id="AlertNotifierWorker-0", workload_pool=""

0 Karma

PickleRick
SplunkTrust
SplunkTrust

1. Please, don't use real-time alerts. Except for very very rare cases real-time searches should be avoided altogether.

2. How did you confirm that the alerts weren't triggered? Is the email the only action you have defined? Maybe the problem is with your action, not the alert itself.

0 Karma

Kareem_Naeem
Loves-to-Learn

email was the only action 

and how did i know is by checking the triggered alerts secion under activity, and as i said i tried scueduled alerts the same issue 

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

You must explicitly add an action to add an alert to triggered alerts. So if your only action was email, you must check why the email wasn't delivered (look in _internal for sendemail.py). At first glance your logs suggest that the alert notifier was actually dispatched.

0 Karma

Kareem_Naeem
Loves-to-Learn

i have had an error in the email logs and i have fized it and now im receiving emails correctly in my local mail server but still the alert when triggered it's not shown in the triggered alerts 

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Announcing Modern Navigation: A New Era of Splunk User Experience

We are excited to introduce the Modern Navigation feature in the Splunk Platform, available to both cloud and ...

Modernize your Splunk Apps – Introducing Python 3.13 in Splunk

We are excited to announce that the upcoming releases of Splunk Enterprise 10.2.x and Splunk Cloud Platform ...

Step into “Hunt the Insider: An Splunk ES Premier Mystery” to catch a cybercriminal ...

After a whole week of being on call, you fell asleep on your keyboard, and you hit a sequence of buttons that ...