perfmon PercentProcessorTime incorrect maximum

[robertlabrie]

Splunk 6.1 running on Windows 2012 R1 perfmon remote machine Windows 2008 R1.

Querying PercentProcessorTime from PerfProc_Process, processes max out at 100%, but this is not correct. Multi-threaded processes can span multiple CPUs, and the theoretical maximum value for this counter is 100 * total CPU cores. Watching % Processor Time in pefmon, I see an instance of sqlserver cross 400%, which is expected, since this represents 25% of my 16 core server.

1) Am I using the wrong counter/query?
2) Is this a known issue with this version of Splunk?
3) Is it a bug?

From powershell (zero valued entries removed)

    PS C:\Users\admin.rlabrie> Get-Counter "\Process(*)\% Processor Time"

    Timestamp                 CounterSamples                                                                                      
    ---------                 -------------- 

    10/13/2015 8:06:21 AM     \\loris-sqlb\process(idle)\% processor time :                                                       
                              1184.87433030096                                                                                    
                              \\loris-sqlb\process(sqlservr)\% processor time :                                                   
                              179.805832224659                                                                                    
                              \\loris-sqlb\process(w3wp)\% processor time :                                                       
                              152.143396497789                                                                                    
                              \\loris-sqlb\process(_total)\% processor time :                                                     
                              1578.29563841645                                                                                    

[ramdaspr]

Apparently that is a known caveat. AFAIK this exists in 6.1.x since I have run into the same problem. Not sure if this is resolved in 6.2 since the docs don't specifically call this out.

Quoting the docs

Due to how Microsoft tallies CPU usage with the Processor:% Processor Time and Process:% Processor Time counters, these counters do not return a value of more than 100 regardless of the number of CPUs or cores in the system.
This is by design - these counters subtract the amount of time spent on the Idle process from 100%.

[robertlabrie]

Thanks for your reply. Yes, they still have it listed as a caveat in the 6.2 and 6.3 docs. Trouble is, it doesn't fit with the powershell snippet I posted above, which clearly shows counters above 100%. Thanks though.

[ramdaspr]

PerfMon in PowerShell actually considers the number of cores in the system while pulling data from Win32_PerfRawData_PerfProc_Process while reporting.

Win32_PerfFormattedData_PerfProc_Process on the other hand does not and is capped at 100% even though the process threads can be across multiple cores and sum up greater than 100%. This the same class which splunk for windows uses which is why the default splunk setup is maxes out at 100%.

[robertlabrie]

Thanks Ramdaspr, that helps. So when I setup a perfmon input, it's just wrapping some WMI around it and calling Win32_PerfFormattedData_XXXX? How can I get Win32_PerfRawData_XXXX and cook it myself at search time? I want this: https://msdn.microsoft.com/en-us/library/ms974615.aspx

[ramdaspr]

Yes, thats pretty much how it works.

Instead of manually calculating via PerfRawData you might have an easier life using Splunk Add-on for Microsoft PowerShell and running the powershell script to provide you the inputs you know are reliably replicated. I havent actually tried implementing this myself so YMMV.