topic Re: How do I subtotal processor utilization? in Splunk Search

How do I subtotal processor utilization?

NickJLange — Sun, 10 Apr 2016 16:09:57 GMT

Disclaimer: I'm not saying this particular example is useful analysis - I'm just not sure how to think about solving a problem like this in Splunk properly.

I have thousands of events of Zabbix Data where socket-wide data points are normalized into individual events. i.e. system.cpu.util[socket,core,type] across heterogeneous hardware configurations (i.e. # of sockets or # of cores are different).
I want to understand the distribution of the load across a socket by machine modeltype to ensure it matches up to temperature readings - and then flag outliers (either on temperature or idle cores).

I've seen tricks around extracting the itemKey into named Variables which I think works because the timestamp is exactly the same.... but how do you run stats on variables that might not exist? (i.e. socket 4 or core 20?)

Does any of this make sense?

Re: How do I subtotal processor utilization?

somesoni2 — Mon, 11 Apr 2016 15:20:23 GMT

Is list of possible socket/core fixed?

Re: How do I subtotal processor utilization?

martin_mueller — Mon, 11 Apr 2016 21:39:14 GMT

Do provide some sample data.

Re: How do I subtotal processor utilization?

NickJLange — Sat, 16 Apr 2016 17:54:03 GMT

It is hard to predict the socket/core count... but it is a finite set.

Re: How do I subtotal processor utilization?

NickJLange — Tue, 29 Sep 2020 09:25:45 GMT

It's not very exciting (one row per pseudo-event):

_time,host,itemKey="system.cpu.util[user_utilization,#socket,#core,]",value=int
....
_time,hostN,itemKey="system.cpu.util[user_utilization,#socket,#core,]",value=int

Re: How do I subtotal processor utilization?

NickJLange — Mon, 18 Apr 2016 12:27:16 GMT

Currently, the query uses rex to extract the #socket/#core are extracted to new variables via Rex...

Re: How do I subtotal processor utilization?

somesoni2 — Mon, 18 Apr 2016 13:51:24 GMT

What will the field value contains?

Re: How do I subtotal processor utilization?

NickJLange — Fri, 06 May 2016 15:09:59 GMT

an integer value from 1- 100. representing utilization ... the equiv of /proc/stat

Re: How do I subtotal processor utilization?

jkat54 — Sat, 07 May 2016 00:56:26 GMT

  ... host=hostname |eval socket=if(isnull(socket),"null",socket) |  timechart avg(value) max(value) by socket

AND

  ... host=hostname | eval core=if(isnull(core),"null",core)| timechart avg(value) max(value) by core

should be fine for a host by host basis. Both would work well on a dashboard with a drop down list to select the hostname etc.

 ... |eval socket=if(isnull(socket),"null",socket) | eval core=if(isnull(core),"null",core)| stats avg(value) max(value) by host core socket

The above should be fine for an analyst to select specific time ranges with time picker and see if activity spikes occured, etc.

Re: How do I subtotal processor utilization?

NickJLange — Sun, 08 May 2016 15:18:32 GMT

Thank you for the helpful suggestion. I'm looking for more aggregate trends across a class of hosts with different underlying hardware models - which sort of precludes individual host analysis with eyeballs...