I have 2 Splunk servers with the same alert on both of them.
One is triggering the alert and the other one is not.
Even by configuring the most basic alert search :
|noop|stats count|eval count = count + 1
which returns 1 line with count 1
One of them triggers and the other won't even though they are the same, so I think that there is something else outside the alert configuration.
What can I verify?
In later versions of Splunk (not sure when it changed), you have to use the
Add Actions button and select the
Add to Triggered Alerts alert action. Perhaps one of your Search Heads is an older version of Splunk or perhaps it is configured in such a way that this action is always auto-added. In any case, you should be able to manually add this to the ones that don't have it.
By the alert not trigger, what exactly do you mean? If you look at jobs, does it show up there? Or just that, say, it won't send you an email? Because for the latter I'd check for differences in email setup between the two, AND check that whatever you are relaying through has allowed both servers to do this.
I mean that I don't see it in Activity -> Triggered Alerts, and neither in the bottom of the page of the alert where it says "There are no fired events for this alert."
The other alert does appear on both the places though on the second Splunk.