topic Re: perfmon PercentProcessorTime incorrect maximum in Splunk Search

perfmon PercentProcessorTime incorrect maximum

robertlabrie — Mon, 12 Oct 2015 21:04:10 GMT

Splunk 6.1 running on Windows 2012 R1 perfmon remote machine Windows 2008 R1.

Querying PercentProcessorTime from PerfProc_Process, processes max out at 100%, but this is not correct. Multi-threaded processes can span multiple CPUs, and the theoretical maximum value for this counter is 100 * total CPU cores. Watching % Processor Time in pefmon, I see an instance of sqlserver cross 400%, which is expected, since this represents 25% of my 16 core server.

1) Am I using the wrong counter/query?
2) Is this a known issue with this version of Splunk?
3) Is it a bug?

From powershell (zero valued entries removed)

    PS C:\Users\admin.rlabrie> Get-Counter "\Process(*)\% Processor Time"

    Timestamp                 CounterSamples                                                                                      
    ---------                 -------------- 

    10/13/2015 8:06:21 AM     \\loris-sqlb\process(idle)\% processor time :                                                       
                              1184.87433030096                                                                                    
                              \\loris-sqlb\process(sqlservr)\% processor time :                                                   
                              179.805832224659                                                                                    
                              \\loris-sqlb\process(w3wp)\% processor time :                                                       
                              152.143396497789                                                                                    
                              \\loris-sqlb\process(_total)\% processor time :                                                     
                              1578.29563841645

Re: perfmon PercentProcessorTime incorrect maximum

ramdaspr — Tue, 13 Oct 2015 03:41:38 GMT

Apparently that is a known caveat. AFAIK this exists in 6.1.x since I have run into the same problem. Not sure if this is resolved in 6.2 since the docs don't specifically call this out.

Quoting the docs

Due to how Microsoft tallies CPU usage with the Processor:% Processor Time and Process:% Processor Time counters, these counters do not return a value of more than 100 regardless of the number of CPUs or cores in the system.
This is by design - these counters subtract the amount of time spent on the Idle process from 100%.

Re: perfmon PercentProcessorTime incorrect maximum

robertlabrie — Tue, 13 Oct 2015 12:39:40 GMT

Thanks for your reply. Yes, they still have it listed as a caveat in the 6.2 and 6.3 docs. Trouble is, it doesn't fit with the powershell snippet I posted above, which clearly shows counters above 100%. Thanks though.

Re: perfmon PercentProcessorTime incorrect maximum

ramdaspr — Tue, 29 Sep 2020 07:34:27 GMT

PerfMon in PowerShell actually considers the number of cores in the system while pulling data from Win32_PerfRawData_PerfProc_Process while reporting.

Win32_PerfFormattedData_PerfProc_Process on the other hand does not and is capped at 100% even though the process threads can be across multiple cores and sum up greater than 100%. This the same class which splunk for windows uses which is why the default splunk setup is maxes out at 100%.

Re: perfmon PercentProcessorTime incorrect maximum

robertlabrie — Tue, 29 Sep 2020 07:34:33 GMT

Thanks Ramdaspr, that helps. So when I setup a perfmon input, it's just wrapping some WMI around it and calling Win32_PerfFormattedData_XXXX? How can I get Win32_PerfRawData_XXXX and cook it myself at search time? I want this: https://msdn.microsoft.com/en-us/library/ms974615.aspx

Re: perfmon PercentProcessorTime incorrect maximum

ramdaspr — Fri, 16 Oct 2015 00:13:38 GMT

Yes, thats pretty much how it works.

Instead of manually calculating via PerfRawData you might have an easier life using Splunk Add-on for Microsoft PowerShell and running the powershell script to provide you the inputs you know are reliably replicated. I havent actually tried implementing this myself so YMMV.