Why am I seeing CPU Usage discrepancy between S.o.S - Splunk on Splunk and PerfMon?

[nk-1]

Splunk-on-Splunk (SoS) indicates 80% - 100% CPU usage, due to searches:

PerfMon indicates about 25% CPU usage:

Why the discrepancy?

[hexx · Accepted solution]

I'm not sure which perfmon counter you are querying, but my best guess here is that it is a counter that expresses CPU usage as a percentage of the total amount of CPU resources on the system.

In contrast, S.o.S expresses CPU usage per process class (the "searches" class including all search processes) as a percentage of one CPU core.

If my theory is correct, you have a 4-core machine here where one or two searches are running and consuming ~ 1 CPU cores altogether. Therefore, the CPU usage of searches can be expressed as 25% of all CPU resources (1 out of 4 available CPU cores) or ~ 1 CPU core / 100% of 1 CPU core.

Did I guess correctly?

[nk-1]

You are correct.
Thanks!

Using

[perfmon://CPU Load]
counters = % Processor Time;% User Time
instances = _Total
interval = 10
object = Processor
index = wmi