Splunk Alerts failing to Trigger

[alexspunkshell]

I have a scheduled alert running every 15 minutes in the cron schedule.

I set trigger action as Email, ServiceNow ticket & MS Teams notification.

Here 80% of the alerts I am receiving successfully. But i am failing to receive the remaining 20% alerts in Email, ServiceNow tickets & MS Teams.

But when I am running the search I can able to find the result but I didn't receive the same alerts.

When I search scheduler logs  I didn't find any failure logs.

Please help here.

[alexspunkshell]

@danielcj Thanks for your reply.

How is your alert defined? - Number of results greater than 0

I see only "status=Done" in  View Recent. I didn't see my failed alerts here.

Below is the screenshot of the alert.

[isoutamo]

Hi

it seems that you have added Alert Throttling here. This means that it didn't fire again same alert within Suppress triggering for time, which you have 7 days. Can this be the reason for no fire alerts?

r. Ismo

https://docs.splunk.com/Documentation/SplunkCloud/latest/Alert/ThrottleAlerts

[alexspunkshell]

@isoutamo I disabled the throttle now. But again the same issue persists.

When I check the index=_internal & scheduler logs it is showing the status as success. Whereas I didn't receive any alert ServiceNow/Email/MS teams.

Out of 10 alerts, I am receiving 8 alerts properly. 2 alerts always failing.

[danielcj]

Hello,

How is your alert defined? Verify the Trigger Conditions and make sure that these configs are correct.

You can use the schedule options: Once OR For each result.

If your alert return multiple results and you need to send an action for each result select the For each result option, select Once otherwise. 

You can view the recent results of your scheduled alert on "Settings > Searches, Reports, and Alerts > Filter your alert > click on View Recent" for further troubleshooting.

Thanks.