Splunk Search

How to chart raw Windows perfmon values over time?

tmarlette
Motivator

I'm attempting to chart some raw windows perfmon values on a chart over time, and I can't seem to find a way. I've been surfing here for a bit, but I haven't seen anything, so i figured I would ask.

This is the search that I used:

sourcetype=perfmon* collection=Process instance=splunkd counter="% Processor Time" | timechart avg(Value) by host

The thing is i'm not looking for an 'average' of the values, i'm just looking to chart the raw values over time. I don't need an avg / sum / max / min function, just how to plot the raw data.
I've attempted this as well:

sourcetype=perfmon* collection=Process instance=splunkd counter="% Processor Time" | timechart Value by host

but splunk says "Error in 'timechart' command: The specifier 'Value' is invalid. It must be in form (). For example: max(size)."

Is there a way to do this without the function?

0 Karma
1 Solution

somesoni2
Revered Legend

Just try this

sourcetype=perfmon* collection=Process instance=splunkd counter="% Processor Time" | table _time host Value

There is a limit on no of points that can be displayed on the chart.

Also, try this

sourcetype=perfmon* collection=Process instance=splunkd counter="% Processor Time" | table _time host Value | xyseries _time host Value

View solution in original post

hagjos43
Contributor

This also works (add host values as needed):

 sourcetype="Perfmon:CPU Load" | bucket _time span=1m | stats avg(Value) by _time

And will plot an average line as well.

Source: http://gosplunk.com/average-cpu-usage/

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Charting the values as they are using plain xyseries is a bit troublesome because you will not get a linear x-axis. If there's a gap in your data that gap will be invisible, if there's times with higher sampling frequency those times will be stretched.

If you're unhappy with | timechart avg(Value) by host because it flattens your data too much, consider increasing the number of data points like this: | timechart bins=500 avg(Value) by host
If you're unhappy with the average consider median(Value) or max(Value) or p95(Value) depending on your needs.

thomrs
Communicator

You have to specify an aggregator or function with timechart.

Give xyseries that should do what you need.

look http://docs.splunk.com/Documentation/Splunk/6.2.1/SearchReference/Xyseries

somesoni2
Revered Legend

Just try this

sourcetype=perfmon* collection=Process instance=splunkd counter="% Processor Time" | table _time host Value

There is a limit on no of points that can be displayed on the chart.

Also, try this

sourcetype=perfmon* collection=Process instance=splunkd counter="% Processor Time" | table _time host Value | xyseries _time host Value

tmarlette
Motivator

xyseries did the trick, but ill keep using a bunch of bins in mind in order to limit the flattening of the data.

Thanks guys!

0 Karma

ppablo
Retired

Hi @tmartlette

Glad you got a lot of input from other users to find your solution. Don't forget to accept one of the answers to resolve this post!

Patrick

0 Karma

tmarlette
Motivator

I would, but for some reason it won't let me at the moment. shrugs. This is the correct answer that I'm commenting on, though others have some good things to watch for as well.

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...