Alert when service stopped using perf mon

[matthewcanty]

source="Perfmon*" counter="% Processor Time" | chart last(Value) by _time, host

How can I develop this search into an alert, which detects when a host has stopped sending perfmon events.

I would like it to send an email to me stating which host is no longer working.

[cphair · Accepted solution]

If you have an index dedicated to perfmon, you might find metadata easier:

| metadata type=hosts index=perfmon | where lastTime<now()-900 

This search returns the machines that haven't reported anything in 15 minutes; I'm assuming the actual counter is irrelevant to you. You can adjust the time as required, but I wouldn't go much shorter or you'll get a bunch of false positives during reboots. Once you have the search tuned the way you want it, you can set up the alert as described in http://docs.splunk.com/Documentation/Splunk/latest/User/SchedulingSavedSearches.