topic Re: Average CPU Usage in Splunk Search

Average CPU Usage

mark_chuman — Mon, 28 Sep 2020 18:25:29 GMT

Trying to create a useful CPU usage search, but coming up short. This search does not reflect what is actually being used for some reason:

index=perfstats host=servername | timechart avg("%_Privileged_Time") by host

For example, when I compare to what the server is actually using to what the search is reporting, the search is coming back extremely low and doesn't match up with actual usage.

Re: Average CPU Usage

MuS — Thu, 11 Dec 2014 14:59:40 GMT

Hi mark.chuman,

timechart will do aggregation on the event values, if you don't need/want that to happen use stats or chart like this:

index=perfstats host=servername | stats avg("%_Privileged_Time") AS avg_PrivTime max("%_Privileged_Time") AS max_PrivTime min("%_Privileged_Time") AS min_PrivTime by host, _time

index=perfstats host=servername | chart avg("%_Privileged_Time") AS avg_PrivTime max("%_Privileged_Time") AS max_PrivTime min("%_Privileged_Time") AS min_PrivTime over _time by host

Both results are basically the same like using timechart, but without the aggregation.

hope this helps ...

cheers, MuS

Re: Average CPU Usage

mark_chuman — Mon, 28 Sep 2020 18:25:34 GMT

many thanks for this and it's a big help. I'm able to use this search to effectively track the average CPU consumption for the servers, but when I say trend it out over time (7+ days for example), it doesn't look very clean as in the trend lines are actually lines, they appear more as bars.

search - index=perfstats host=servername | chart avg("%_Processor_Time") over _time by host

I'll post an image..

Re: Average CPU Usage

mark_chuman — Thu, 11 Dec 2014 15:34:16 GMT

not enough karma to upload image 🙂

Re: Average CPU Usage

mark_chuman — Mon, 28 Sep 2020 18:25:37 GMT

but for example, this search comes back very clean in the display

index=perfstats host=servername | timechart avg("%_Idle_Time") by host

Re: Average CPU Usage

MuS — Thu, 11 Dec 2014 17:05:19 GMT

sure this is why chartdoes not aggregate like timechart does and as another pointer, did your try other formats for your chart? You can also change the behavior of the lines for example no results (connect, treat as zero ...)

Re: Average CPU Usage

MuS — Thu, 11 Dec 2014 17:06:15 GMT

you can link any picture web page like tiny url, no need to upload a picture 😉

Re: Average CPU Usage

mark_chuman — Mon, 28 Sep 2020 18:25:39 GMT

Almost there I think.

So, I adjusted the visualization (format, connect the dots basically), but it's weird as there is what looks to be a true CPU metric followed by a very low, inaccurate metric which makes the trend line extremely spiky and doesn't really resemble what the CPU is doing.

So, the search is pretty simple - index=perfstats host=eivc201vwin | chart avg("%_Processor_Time") over _time by host

This image shows how spiky the trend line is - http://pasteboard.co/2mtXDwFl.png

This image shows the actual sample data where you see this "piggy back" type metric - http://pbrd.co/16bamV9

You'll see a reading of say 45 followed immediately by a real low reading of 3.

Thanks again for your time!

Re: Average CPU Usage

MuS — Thu, 11 Dec 2014 18:16:33 GMT

You're running the script to get the cpu stats every 5min right? Your spiky graph caused by this if you decrease the interval and get more results the smoother the graph will be

Re: Average CPU Usage

mark_chuman — Mon, 15 Dec 2014 14:16:31 GMT

It is grabbing the stats every 5 minutes, but it's weird as there are two stats for every 5 minute pull. They are a second apart and these are the metrics that appear to be making the trend spiky as the second metric is always much lower than the first. For example, a metric comes in at say 9:01:25 for 45% (or 45.87326) and then a metric comes in at 9:01:26 for say 2% (or 2.34344) and then the cycle starts again at 9:06:25.