Generating an alert for a process running on two hosts at the same time

[justincoon]

We have a service (process) that should only ever be running on one server at a time. We have MS failover clustering setup to manage this but every once in a while someone unintentionally starts it on both servers at the same time. So we want to setup an alert if these processes ever run on more than one host.

I feel like this should be a straightforward query, get the unique count of hosts running the process every minute and alert if it's greater than 1 for more than two minutes... would be one way, but I don't know how to set this up.

[adonio · Accepted solution]

first verify you have the relevant data from both hosts
now try something like this:
... search ... process="YOUR PROCESS" (host="HOST 1" OR host="HOST 2")
| bin _time span=1m
| stats dc(host) as unique_hosts by process _time
| where unique_hosts > 1

[justincoon]

I tested this out in our environment and it looks like it's working, thank you!