On a Windows 2008 R2 server, I've been comparing the %Processor Time counter on the Total instance from Perfmon with the PercentProcessorTime counter on the _Total instance of the PerfOSProcessor object in Splunk. The output from the counters is never the same. In Splunk, I created a separate index and started sampling at one second intervals to see if I could get the two counters to line up but they don't. I'm running version 4.2.1 build 98164. Can someone help me discovery why there is a discrepancy?
I would suspect that unless the counters happened to kick off at the exact same time there could be a discrepancy in %Processor Time. If you were to look at the averages using timechart over a reasonable period of time, do the graphs align?
Sorry, I was being a little to granular with the problem description. The values returned are not even close. For example, Splunk might report a value of 58 PercentProcessorTime and I'll look to see if Perfmon reports that value within say 10 seconds of when Splunk reports it and it never happens.
This same behavior is occurring on all the machines we are monitoring this way. They are all physical machines. I can't tell you that Splunk ever reported the correct values. I just know that at review meetings engineers are getting freaked out when they see the numbers Splunk is reporting.