Linux/Unix App total usage with multiple CPUs

[sherbuckap]

Hi, so I have a question regarding CPU usage and virtualization/multiple processors.

My environment:
Red hat 6 Virtual Machine
2 CPU
8 GB RAM

I was testing some basic alerting using the *Nix Addon for Splunk. The app includes a cpu monitor script out of the box using top. A snag I hit was such: If I monitor for a process taking up 90%+ CPU over a time period that doesn't necessarily mean the VM is capped/flatlined on CPU. For a 2 CPU system in this scenario it's possible that only ~70% of total CPU is being used (90% - 100% of one CPU + what the rest of the processes are using on the second processor). This would all be dependent on how an application runs, how many processors are on the box, etc.

I'm trying to find a more reliable way to monitor total usage. Has anyone had experiences similar to this? I've been giving:

index=os sourcetype=cpu host=SplunkLab03  | multikv fields pctIdle  | where pctIdle<10

A shot and it seems to work well but would appreciate a verification before moving forward.

The script I use to run up cpu is:

while true;
do true;
done

[yuanliu]

On a multicore system, load is roughly additive. So I would simply add values from all CPUs ( sum(pctIdle) by _time, then divide by number of CPUs ( | eventstats dc(CPU)). On the other hand, the sar utility on Linux that sourcetype cpu uses includes the aggregation CPU="all". This saves you from the burden (and complicates the filter if you are to calculate on your own). Hence,

index=os sourcetype=cpu host=SplunkLab03 CPU=all
| where pctIdle<10