and so I can easily enough write the following search
source="perfmon:cpu" | timechart span=15m avg(Value) as CPU
The problem I perceive is that I think permon:cpu is an instantaneous value. If so then the rate at which the agent reports sends updates (or events) sets a minimum detectable load. Basically I'm saying that if I am only getting a cpu event every 5 min, then chances are I would never see a 30sec spike in cpu utilization. I could just up the cpu event rate to someting like 10 sec (to detect 30 sec spikes), but I am wondering if there is another approach that will not involve increasing the number of cpu events dramatically?
You are correct in your approach, but if you want the granular info you need to see the perfmon counter more frequently i.e. the interval needs to be in seconds. And you should not be concerned if there is a CPU spike for a second or two. Monitoring perfmon _total instance per day wont consume much from your license volume.
Second approach is an indirectly get the average value of your perfmon counter using one script. For that you need to collect instantaneous data into csv file using windows data collector and rather than perfmon.conf , in INPUTS.CONF you trigger a script which calculates the value for you and send an average for that 5 mins or whatever interval you want so that you don't loose anything.