Splunk-on-Splunk (SoS) indicates 80% - 100% CPU usage, due to searches:
PerfMon indicates about 25% CPU usage:
Why the discrepancy?
I'm not sure which perfmon counter you are querying, but my best guess here is that it is a counter that expresses CPU usage as a percentage of the total amount of CPU resources on the system.
In contrast, S.o.S expresses CPU usage per process class (the "searches" class including all search processes) as a percentage of one CPU core.
If my theory is correct, you have a 4-core machine here where one or two searches are running and consuming ~ 1 CPU cores altogether. Therefore, the CPU usage of searches can be expressed as 25% of all CPU resources (1 out of 4 available CPU cores) or ~ 1 CPU core / 100% of 1 CPU core.
Did I guess correctly?
I'm not sure which perfmon counter you are querying, but my best guess here is that it is a counter that expresses CPU usage as a percentage of the total amount of CPU resources on the system.
In contrast, S.o.S expresses CPU usage per process class (the "searches" class including all search processes) as a percentage of one CPU core.
If my theory is correct, you have a 4-core machine here where one or two searches are running and consuming ~ 1 CPU cores altogether. Therefore, the CPU usage of searches can be expressed as 25% of all CPU resources (1 out of 4 available CPU cores) or ~ 1 CPU core / 100% of 1 CPU core.
Did I guess correctly?
You are correct.
Thanks!
Using
[perfmon://CPU Load]
counters = % Processor Time;% User Time
instances = _Total
interval = 10
object = Processor
index = wmi