topic Re: Splunk alert trigger issue in Alerting

Splunk alert trigger issue

Kareem_Naeem — Mon, 14 Oct 2024 17:29:46 GMT

my alert is not triggered even with many matching events here are the details:

while the activity that generate these logs is running the real time alert is processing and found those events in the screenshot. i have waited for 5 mins and same issue

i have also tries scheduled and still the same issue no triggering

Re: Splunk alert trigger issue

PickleRick — Mon, 14 Oct 2024 17:43:56 GMT

1. Please, don't use real-time alerts. Except for very very rare cases real-time searches should be avoided altogether.

2. How did you confirm that the alerts weren't triggered? Is the email the only action you have defined? Maybe the problem is with your action, not the alert itself.

Re: Splunk alert trigger issue

Kareem_Naeem — Mon, 14 Oct 2024 17:51:06 GMT

email was the only action

and how did i know is by checking the triggered alerts secion under activity, and as i said i tried scueduled alerts the same issue

Re: Splunk alert trigger issue

Kareem_Naeem — Mon, 14 Oct 2024 18:22:10 GMT

and here i have tried the scheduled alert and here is the logs related to that alert:

log 1:

10-14-2024 22:16:01.088 +0400 INFO SavedSplunker - AlertNotifier::notifySearchCompleted: called for sid=scheduler__kareem__search__kareem_at_1728929760_35, condition=1

log 2:

10-14-2024 22:16:01.809 +0400 INFO SavedSplunker - savedsearch_id="nobody;search;kareem", search_type="scheduled", search_streaming=0, user="kareem", app="search", savedsearch_name="kareem", priority=default, status=success, digest_mode=1, durable_cursor=0, scheduled_time=1728929760, window_time=0, dispatch_time=1728929760, run_time=0.178, result_count=1531, alert_actions="email", sid="scheduler__kareem__search__kareem_at_1728929760_35", suppressed=0, action_time_ms=718, thread_id="AlertNotifierWorker-0", workload_pool=""

Re: Splunk alert trigger issue

PickleRick — Mon, 14 Oct 2024 18:43:50 GMT

You must explicitly add an action to add an alert to triggered alerts. So if your only action was email, you must check why the email wasn't delivered (look in _internal for sendemail.py). At first glance your logs suggest that the alert notifier was actually dispatched.

Re: Splunk alert trigger issue

Kareem_Naeem — Tue, 15 Oct 2024 20:49:48 GMT

i have had an error in the email logs and i have fized it and now im receiving emails correctly in my local mail server but still the alert when triggered it's not shown in the triggered alerts