Given sourcetype=ps and sourcetype=top , in both of which pctCPU are present, how do I associate pctCPU from top only while using fields unique to ps? (Despite identical field name, values in these two sources represent very different things.)
In Splunk Add-on for Nix, for example, *ps and top both contain fields PID , COMMAND and pctCPU . (They share some other field names of interest which I will not use in this example.) As @Paolo Prigione pointed out many years ago, pctCPU in ps is not useful for monitoring. (https://answers.splunk.com/answers/27398/is-nix-sourcetype-ps-pctcpu-really-suitable-for-charting-ootb.html) In the simplest use case, pctCPU in top would give the instantaneous CPU usage of each process. However, COMMAND in top only gives a simple program name, which is insufficient for my purposes. (In the old nix for Splunk, *ps' COMMAND includes full arguments; in Splunk Add-on for Nix, *ps has a separate ARGS field.)
Conceivably I can associate top's pctCPU values with ps' app (combination of COMMAND and ARGS in the new Splunk Add-on for Nix) by joining a *top search with a ps search. This looks very wasteful, however. So I thought I would tackle it by a simple search, then eliminate values from ps.
index=os (sourcetype=ps OR sourcetype=top)
| bucket _time span=1m
| stats values(if(sourcetype="ps",app,COMMAND)) as app values(eval(if(sourcetype="top",pctCPU,null()))) as pctCPU by _time PID
( bucket _time is necessary because, though launched with the same frequency, the two sources often have sub-minute stagger.) This works for all processes output from ps. However, as ps and top do not always survey the same processes even when they are launched within a subsecond, some processes captured by ps will not show in top of the same time interval, and vice versa. As a result, the above strategy gives null values when the process is in ps only. I want to fill these gaps with values from ps, because for these extremely momentary processes, pctCPU from ps has the same significance as that from top.
In other words, I want eliminate value of pctCPU from ps when top is available, but use value from ps when not. (The first term in the example, values(if(sourcetype="ps",app,COMMAND)) as app , is a much more sophisticated macro output in reality. That output can cause gaps when a process is only in ps but missing from top.)
... View more
