Monitoring Splunk

Why is the CPU usage showing 80-100% (falsely)? -- Search help

lbalaur
Explorer

I am trying to get CPU usage for a specific process in windows. My search looks like this:

host=host1 AND sourcetype="Perfmon:Process"  AND  counter="% Processor Time" AND process_name="server*" | table _time, counter, process_name, Value

My result is showing mostly 100 for Value which is not really true. Windows runs on VM.

Result looks like this:

2017-09-22T14:40:28.000-0400    % Processor Time    server  100
2017-09-22T14:39:43.000-0400    % Processor Time    server  100
2017-09-22T14:37:28.000-0400    % Processor Time    server  100
2017-09-22T14:32:58.000-0400    % Processor Time    server#1    100
2017-09-22T14:32:13.000-0400    % Processor Time    server  100
2017-09-22T14:38:13.000-0400    % Processor Time    server  100
2017-09-22T14:31:28.000-0400    % Processor Time    server#1    11.30968265
2017-09-22T14:21:43.000-0400    % Processor Time    server  100
2017-09-22T14:18:43.000-0400    % Processor Time    server#1    0.105369743
2017-09-22T14:36:43.000-0400    % Processor Time    server  0.034732856
2017-09-22T14:35:58.000-0400    % Processor Time    server#1    0.14049302
2017-09-22T14:29:13.000-0400    % Processor Time    server  100
2017-09-22T14:28:28.000-0400    % Processor Time    server#1    84.84122861
2017-09-22T14:20:58.000-0400    % Processor Time    server#1    100
2017-09-22T14:16:28.000-0400    % Processor Time    server  100
2017-09-22T14:14:58.000-0400    % Processor Time    server#1    100

What should I do? Why is it pulling all the 100s? 80% of events show a 100. Is it an agent config issue?

0 Karma

tmarlette
Motivator

Perfmon is not very intuitive when watching processor utilization per process. I'm going to assume you're watching process utilization on a multi-processor machine. you may want to use 'System: %Total Processor Time' instead.

Here's some documentation:
https://docs.microsoft.com/en-us/sql/relational-databases/performance-monitor/monitor-cpu-usage

If you are monitoring a multi-processor machine, and if you're using this counter to monitor CPU utilization, then you're only watching a single thread. generally speaking, 1 thread = 1 core of the machine, so on a 16 core machine 1 core at 100%, is quite normal. When you have 10 instances of 'server#1' running at 100%, you would likely see performance degradation.

natalienguyen
Explorer

thanks @tmarlette, I was having the same issue and the %Total Processor Time helped!

0 Karma

lbalaur
Explorer

If System: %Total Processor Time is not showing up in splunk - I need to ask my admins to add this statistic then?

0 Karma

tmarlette
Motivator

Yeah, you would need to as your Splunk admins to add that counter in the Splunk_TA_windows app.

0 Karma

Sukisen1981
Champion

hi - cpu usage (or utilization) has to be over time, so what happens if you try something like this-
host=host1 AND sourcetype="Perfmon:Process" AND counter="% Processor Time" AND process_name="server*" | timechart avg(value)
OR
host=host1 AND sourcetype="Perfmon:Process" AND counter="% Processor Time" AND process_name="server*" | timechart avg(value) by process_name

if this works and you just need other fields,....you just need to eval them AFTER the timechart command or use a simple appendcols in this case

0 Karma

lbalaur
Explorer

If you do a timechart how are all those false 100s going to affect your average? My concern is not building the chart but underlying data. Based on my observations usage never reached a 100% but splunk keeps bringing it in.
I tested the timechart, btw, and it's staying above 50% on average when in fact it should be around 10%

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In January, the Splunk Threat Research Team had one release of new security content via the Splunk ES Content ...

Expert Tips from Splunk Professional Services, Ensuring Compliance, and More New ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Observability Release Update: AI Assistant, AppD + Observability Cloud Integrations & ...

This month’s releases across the Splunk Observability portfolio deliver earlier detection and faster ...