See if these help:
I believe you may be experiencing schedule skewing.
I have configured Slack and email for the same alert, however we are not getting the same number of alerts in slack, alert has been scheduled to run every 2 minutes. Email alerts are coming properly but not in Slack. What could be an issue?
And there is also a delay in alerts that are coming on Slack. Quick help would be appreciated.
Are you sure you are not hitting Slack's API rate limit?
Limits are applied to workspaces, so if you have several integrations (in yours and other channels) you could find that slack is limiting the number of messages delivered.
Before I go ahead and accept this answer, need to know few things.
I am encountering the same issue again, alerts are coming to slack channel but there is a delay of 30 mins. For instance, if alert has been scheduled to run every 5 mins, In 30 mins I am getting only 2 alerts. Is it because of network issue?
One more thing, I would like to highlight here is that, webhook created by me is now not visible under "Manage Webhook" page but still alerts are coming into splunk with the delay of 60 mins - 90 mins.
Have you tried using curl from the server to post to slack?
Maybe there is packet loss or very slow connection to slack.
Or have you tried looking at the scheduler logs?
Have you tried the job inspector to see if search.log has any errors when the alertS run?
We are getting alerts in slack but there is a delay in time, suppose I am scheculing an alert for 2 minutes for email and slack, in next 6 minutes I get 3 email alerts but only 2 slack alerts.
And in splunkd.log I see this error message "Alert action script returned error code=255". What could be an issue?