Strangely with enough tinkering, we both came up with similar search queries. I ran it and search returned results with hosts that had less than 5 events. For instance, in the last 5 minutes I get a PC with 2 results.
5:16:54 and 5:19:03, both falling within the target usage value. Then I decided to do a query in general for that PC. I have Splunk to query for cpu usage every 60 seconds for about 40 PCs. It also queries every 60 seconds for amount of RAM free, and disk idle time (for those same 40 PCs). To my surprise, I see 12 events for a 15 minute window (a query looking for this PC, in the perfmon index for any entries regarding cpu usage). I should see 15 events (or possibly 14). For instance, 5:16:54 and 5:14:21 are the last two events. Why is there over a 2 minute gap? splunkd log doesn't say anything (no errors regarding this PC).
... View more