Splunk Search

Generating an alert for a process running on two hosts at the same time

justincoon
New Member

We have a service (process) that should only ever be running on one server at a time. We have MS failover clustering setup to manage this but every once in a while someone unintentionally starts it on both servers at the same time. So we want to setup an alert if these processes ever run on more than one host.

I feel like this should be a straightforward query, get the unique count of hosts running the process every minute and alert if it's greater than 1 for more than two minutes... would be one way, but I don't know how to set this up.

0 Karma
1 Solution

adonio
Ultra Champion

first verify you have the relevant data from both hosts
now try something like this:
... search ... process="YOUR PROCESS" (host="HOST 1" OR host="HOST 2")
| bin _time span=1m
| stats dc(host) as unique_hosts by process _time
| where unique_hosts > 1

View solution in original post

0 Karma

adonio
Ultra Champion

first verify you have the relevant data from both hosts
now try something like this:
... search ... process="YOUR PROCESS" (host="HOST 1" OR host="HOST 2")
| bin _time span=1m
| stats dc(host) as unique_hosts by process _time
| where unique_hosts > 1

0 Karma

justincoon
New Member

I tested this out in our environment and it looks like it's working, thank you!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...