duplicate RT alerts in Search Head Cluster

[sat94541]

We have an issue with quite a number of duplicate alerts, all but 1 are real time searches. One example there were 5 alerts at the exact same time from 5 of the 11 Search Head in the Search Head Cluster .

Any ideas?

[rbal_splunk]

Here are few things to check:

1) Check if the delegation failed due to quota issues ? i.e with in the same minute we re-try twice..
2) Also, check if you see connectivity issue between search head cluster and Captain. It could be because the SH's running the alerts are not able to connect to captain
3)Check the health of the members during that 5 alert fired period, like is the captain switching too fast.
4) Check if the members are all part of the show shcluster-status and the RT jobs have duplicate among members.When a captain receives information about searches in the cluster and it finds duplicate RT searches they should be cancelled.