Splunk perfmon disk io search

[tkerr1357]

Hello all,

Looking for some help with a perfmon search. 

index=perfmon host=myhost01s* sourcetype="PerfmonMk:LogicalDisk"  instance=_total

| timechart sum(Disk_Transfers/sec) span=90s

This gives me all the IO of the hosts, but it only seems accurate if I chart it to 90s span, which is how often it queries. If I change that then the sum obviously just puts all the values together.

What I want to do is show longer periods of time, but still have it counting the sum of each reading (like a bucket) over time and not adding all numbers over the whole period (span) together. I want to force it to always use the time as the bucket. Any help is much appreciated.

[tscroggins · Accepted solution]

@tkerr1357 

If your perfmon input interval is 90 seconds, then Disk_Transfers = 90 * 'Disk_Transfers/sec'. You can displayed corrected Disk_Transfers/sec and Disk_Transfers values for any time span using this:

index=perfmon host=myhost01s* sourcetype="PerfmonMk:LogicalDisk" instance=_total
| timechart fixedrange=f minspan=90s per_second(eval(90 * 'Disk_Transfers/sec')) as Disk_Transfers/sec sum(eval(90 * 'Disk_Transfers/sec')) as Disk_Transfers by host

If your interval is unknown, you can estimate it from data:

index=perfmon host=myhost01s* sourcetype="PerfmonMk:LogicalDisk" instance=_total
| streamstats current=f last(_time) as last_time by host
| eval interval=_time-last_time
| where NOT isnull(interval)
| timechart fixedrange=f per_second(eval(interval * 'Disk_Transfers/sec')) as Disk_Transfers/sec sum(eval(interval * 'Disk_Transfers/sec')) as Disk_Transfers by host

Note that your tails may not cover an entire span. E.g. If timechart chooses span=1h and your search time range is not snapped to the nearest hour with @h, then your first and last values will be skewed by missing data.