Splunk Search

Chart # of instances for a process, w/ sum of RAM and avg CPU of all those instances

mxanareckless
Path Finder

Really stumped on this. We would like to count the number of instances of each process run on a server, and present the sum of RAM and CPU usage for all those instances of the process.

Here is an example, from the server side, of all the instances of a single app being run, that we would like to aggregate:

 

 

PID USERNAME NLWP PRI NICE  SIZE   RES STATE     TIME    CPU COMMAND
 22661 cacheuse    1  59    0  452M  441M sleep     0:03  0.00% cache
 22664 cacheuse    1  59    0  452M  440M sleep     0:00  0.00% cache
 22669 cacheuse    1  59    0  452M  440M sleep     0:00  0.00% cache
 22667 cacheuse    1  59    0  452M  440M sleep     0:00  0.00% cache
 22665 cacheuse    1  59    0  452M  440M sleep     0:00  0.00% cache
 22670 cacheuse    1  59    0  452M  440M sleep     0:00  0.00% cache
 22668 cacheuse    1  59    0  452M  440M sleep     0:00  0.00% cache
 22666 cacheuse    1  59    0  452M  440M sleep     0:00  0.00% cache
 22953 cacheuse    1  59    0  452M  444M sleep     0:03  0.00% cache
 23053 cacheuse    1  59    0  452M  444M sleep     0:06  0.00% cache
 23052 cacheuse    1  59    0  452M  444M sleep     0:02  0.00% cache
 24543 cacheuse    1  59    0  452M  444M sleep     0:03  0.00% cache
 22941 cacheuse    1  59    0  452M  440M sleep     0:00  0.00% cache
 22945 cacheuse    1  59    0  452M  440M sleep     0:00  0.00% cache
 22944 cacheuse    1  59    0  452M  440M sleep     0:00  0.00% cache
 22943 cacheuse    1  59    0  452M  440M sleep     0:00  0.00% cache
 22946 cacheuse    1  59    0  452M  440M sleep     0:00  0.00% cache
 22942 cacheuse    1  59    0  452M  440M sleep     0:00  0.00% cache
 22947 cacheuse    1  59    0  452M  440M sleep     0:00  0.00% cache
 22940 cacheuse    1  59    0  452M  440M sleep     0:00  0.00% cache
 22948 cacheuse    1  59    0  452M  440M sleep     0:00  0.00% cache
 22939 cacheuse    1  59    0  452M  440M sleep     0:06  0.00% cache
 22938 cacheuse    1  59    0  452M  440M sleep     0:00  0.00% cache
 22671 cacheuse    1  59    0  451M  440M sleep     0:00  0.00% cache
 22663 cacheuse    1  59    0  451M  440M sleep     0:13  0.00% cache
 22662 cacheuse    1  59    0  451M  440M sleep     0:00  0.00% cache
 22649 cacheuse    1  59    0  444M  440M sleep     0:33  0.00% cache
 22932 cacheuse    1  59    0  443M  440M sleep     0:05  0.00% cache
  5863 root       17  59    0  185M  163M sleep   139:37  0.00% sstored
  4570 splunk     43  59    0  177M  130M sleep    15:16  0.00% splunkd

 

 

As you can see, there are 28 instances of the cache program. We would like to roll all of that up into something like this:

Program# instancestotal RAMtotal CPU
cache2812GB0.00%
splunkd1177M0.01%
sstored1185M0.0.1%

For the top sourcetype, the VIRT sourcetype counts RAM in kilobytes.  If VIRT's integer value is greater than 1024, we want the integer multiplied by 1024 and suffixed with the letter "M" for megabytes; and if the integer is greater than 1048576, we want that integer multiplied by 1048576 and suffixed with the letter "G" for gigabytes.
Here is what we've come up with so far, but it's nowhere near what we need:

 

 

index=xxxx sourcetype=top host=xxxx COMMAND!="<n/a>"
| rename COMMAND as Program, pctCPU as "% CPU", USER as User
| regex "% CPU"="(\d+)"
| convert rmunit(VIRT)
| eval inMB=if(VIRT>=1024,1,0), VIRT=floor(if(inMB=1,VIRT/1024,VIRT*1))
| chart sum(VIRT) by Program

 

 

Thank you in advance!

Labels (4)
0 Karma