Deployment Architecture

duplicate RT alerts in Search Head Cluster


We have an issue with quite a number of duplicate alerts, all but 1 are real time searches. One example there were 5 alerts at the exact same time from 5 of the 11 Search Head in the Search Head Cluster .

Any ideas?

Tags (1)
0 Karma

Splunk Employee
Splunk Employee

Here are few things to check:

1) Check if the delegation failed due to quota issues ? i.e with in the same minute we re-try twice..
2) Also, check if you see connectivity issue between search head cluster and Captain. It could be because the SH's running the alerts are not able to connect to captain
3)Check the health of the members during that 5 alert fired period, like is the captain switching too fast.
4) Check if the members are all part of the show shcluster-status and the RT jobs have duplicate among members.When a captain receives information about searches in the cluster and it finds duplicate RT searches they should be cancelled.

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Customer Survey!

If you use Splunk Observability Cloud, we invite you to share your valuable insights with us through a brief ...

Happy CX Day, Splunk Community!

Happy CX Day, Splunk Community! CX stands for Customer Experience, and today, October 3rd, is CX Day — a ...

.conf23 | Get Your Cybersecurity Defense Analyst Certification in Vegas

We’re excited to announce a new Splunk certification exam being released at .conf23! If you’re going to Las ...