We have a service (process) that should only ever be running on one server at a time. We have MS failover clustering setup to manage this but every once in a while someone unintentionally starts it on both servers at the same time. So we want to setup an alert if these processes ever run on more than one host.
I feel like this should be a straightforward query, get the unique count of hosts running the process every minute and alert if it's greater than 1 for more than two minutes... would be one way, but I don't know how to set this up.
first verify you have the relevant data from both hosts
now try something like this: ... search ... process="YOUR PROCESS" (host="HOST 1" OR host="HOST 2")
| bin _time span=1m
| stats dc(host) as unique_hosts by process _time
| where unique_hosts > 1