Splunk Search

How to chart raw Windows perfmon values over time?

tmarlette
Motivator

I'm attempting to chart some raw windows perfmon values on a chart over time, and I can't seem to find a way. I've been surfing here for a bit, but I haven't seen anything, so i figured I would ask.

This is the search that I used:

sourcetype=perfmon* collection=Process instance=splunkd counter="% Processor Time" | timechart avg(Value) by host

The thing is i'm not looking for an 'average' of the values, i'm just looking to chart the raw values over time. I don't need an avg / sum / max / min function, just how to plot the raw data.
I've attempted this as well:

sourcetype=perfmon* collection=Process instance=splunkd counter="% Processor Time" | timechart Value by host

but splunk says "Error in 'timechart' command: The specifier 'Value' is invalid. It must be in form (). For example: max(size)."

Is there a way to do this without the function?

0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

Just try this

sourcetype=perfmon* collection=Process instance=splunkd counter="% Processor Time" | table _time host Value

There is a limit on no of points that can be displayed on the chart.

Also, try this

sourcetype=perfmon* collection=Process instance=splunkd counter="% Processor Time" | table _time host Value | xyseries _time host Value

View solution in original post

hagjos43
Contributor

This also works (add host values as needed):

 sourcetype="Perfmon:CPU Load" | bucket _time span=1m | stats avg(Value) by _time

And will plot an average line as well.

Source: http://gosplunk.com/average-cpu-usage/

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Charting the values as they are using plain xyseries is a bit troublesome because you will not get a linear x-axis. If there's a gap in your data that gap will be invisible, if there's times with higher sampling frequency those times will be stretched.

If you're unhappy with | timechart avg(Value) by host because it flattens your data too much, consider increasing the number of data points like this: | timechart bins=500 avg(Value) by host
If you're unhappy with the average consider median(Value) or max(Value) or p95(Value) depending on your needs.

thomrs
Communicator

You have to specify an aggregator or function with timechart.

Give xyseries that should do what you need.

look http://docs.splunk.com/Documentation/Splunk/6.2.1/SearchReference/Xyseries

somesoni2
SplunkTrust
SplunkTrust

Just try this

sourcetype=perfmon* collection=Process instance=splunkd counter="% Processor Time" | table _time host Value

There is a limit on no of points that can be displayed on the chart.

Also, try this

sourcetype=perfmon* collection=Process instance=splunkd counter="% Processor Time" | table _time host Value | xyseries _time host Value

tmarlette
Motivator

xyseries did the trick, but ill keep using a bunch of bins in mind in order to limit the flattening of the data.

Thanks guys!

0 Karma

ppablo
Retired

Hi @tmartlette

Glad you got a lot of input from other users to find your solution. Don't forget to accept one of the answers to resolve this post!

Patrick

0 Karma

tmarlette
Motivator

I would, but for some reason it won't let me at the moment. shrugs. This is the correct answer that I'm commenting on, though others have some good things to watch for as well.

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...