topic Splunk Alerts failing to Trigger in Alerting

Splunk Alerts failing to Trigger

alexspunkshell — Wed, 25 Aug 2021 04:33:24 GMT

I have a scheduled alert running every 15 minutes in the cron schedule.

I set trigger action as Email, ServiceNow ticket & MS Teams notification.

Here 80% of the alerts I am receiving successfully. But i am failing to receive the remaining 20% alerts in Email, ServiceNow tickets & MS Teams.

But when I am running the search I can able to find the result but I didn't receive the same alerts.

When I search scheduler logs I didn't find any failure logs.

Please help here.

Re: Splunk Alerts failing to Trigger

danielcj — Wed, 25 Aug 2021 05:31:35 GMT

Hello,

How is your alert defined? Verify the Trigger Conditions and make sure that these configs are correct.

You can use the schedule options: Once OR For each result.

If your alert return multiple results and you need to send an action for each result select the For each result option, select Once otherwise.

You can view the recent results of your scheduled alert on "Settings > Searches, Reports, and Alerts > Filter your alert > click on View Recent" for further troubleshooting.

Thanks.

Re: Splunk Alerts failing to Trigger

alexspunkshell — Wed, 25 Aug 2021 05:47:53 GMT

@danielcj Thanks for your reply.

How is your alert defined? - Number of results greater than 0

I see only "status=Done" in View Recent. I didn't see my failed alerts here.

Below is the screenshot of the alert.

Re: Splunk Alerts failing to Trigger

isoutamo — Wed, 25 Aug 2021 06:17:45 GMT

it seems that you have added Alert Throttling here. This means that it didn't fire again same alert within Suppress triggering for time, which you have 7 days. Can this be the reason for no fire alerts?

r. Ismo

https://docs.splunk.com/Documentation/SplunkCloud/latest/Alert/ThrottleAlerts

Re: Splunk Alerts failing to Trigger

alexspunkshell — Fri, 27 Aug 2021 07:42:23 GMT

@isoutamo I disabled the throttle now. But again the same issue persists.

When I check the index=_internal & scheduler logs it is showing the status as success. Whereas I didn't receive any alert ServiceNow/Email/MS teams.

Out of 10 alerts, I am receiving 8 alerts properly. 2 alerts always failing.