Splunk Search
Highlighted

Generating an alert for a process running on two hosts at the same time

New Member

We have a service (process) that should only ever be running on one server at a time. We have MS failover clustering setup to manage this but every once in a while someone unintentionally starts it on both servers at the same time. So we want to setup an alert if these processes ever run on more than one host.

I feel like this should be a straightforward query, get the unique count of hosts running the process every minute and alert if it's greater than 1 for more than two minutes... would be one way, but I don't know how to set this up.

0 Karma
Highlighted

Re: Generating an alert for a process running on two hosts at the same time

SplunkTrust
SplunkTrust

first verify you have the relevant data from both hosts
now try something like this:
... search ... process="YOUR PROCESS" (host="HOST 1" OR host="HOST 2")
| bin _time span=1m
| stats dc(host) as unique_hosts by process _time
| where unique_hosts > 1

View solution in original post

0 Karma
Highlighted

Re: Generating an alert for a process running on two hosts at the same time

New Member

I tested this out in our environment and it looks like it's working, thank you!

0 Karma