Hi all, I am reasoning about the *nix app sourcetype=ps' pctCPU metric and how to plot it correctly.
I see Splunk's nix app generally plots it by doing *... | timechart avg(pctCPU) by ...**. This would be perfectly fine if pctCPU expressed the instantaneous usage of cpu (as top does). Instead, per "man ps" definition (RHEL 5.5):
%cpu %CPU cpu utilization of the process in "##.#" format. Currently, it is the CPU time used divided by the time the process has been running (cputime/realtime ratio), expressed as a percentage. It will not add up to 100% unless you are lucky. (alias pcpu).
pctCPU expresses the average cpu used by the process since its startup (cpu time / total run time)!
Say the process has long been running with low usage, then it has a burst for some minutes, then usage drops again. Ps' pctCPU would not reflect this behaviour as the total cpu time over which it has been computed did not increment that much with respect to total runtime. pctCPU is smoothed in this case.
Does my reasoning make any sense to you?
I have a quite complex solution under work which involves computing deltas of CPUTIME and ELAPSED (splunk's ps.sh definitions) for any multikv'ed ps execution, then compute "instantaneous" pctCPU, then average and plot it.
However, this is a fairly slow (requires to use | sort +host +PID +COMMAND +_time with all its limits) and complex solution.
Has anybody came up with something better?
