topic Re: How to collect perf counters of type average or per second? in Getting Data In

How to collect perf counters of type average or per second?

donhuanmatus — Mon, 28 Sep 2020 11:30:30 GMT

Hi Everybody,

I have a WMI Perf counter query that always returns zero in splunk-wmi.exe for counters with the following types: average and per second.

I run the folowing command:

D:\Splunk\bin>splunk-wmi -wql "SELECT Name, DiskReadBytesPerSec, AvgDiskQueueLength FROM Win32_PerfFormattedData_PerfDisk_PhysicalDisk"

And get the following output:

SPLUNK index= source="WMI:unspecified" sourcetype="WMI:unspecified"

---splunk-wmi-end-of-event---
20120313033326.660213
AvgDiskQueueLength=0
DiskReadBytesPersec=0
Name=0 C: 😧
wmi_type=unspecified

---splunk-wmi-end-of-event---
SPLUNK index= source="WMI:unspecified" sourcetype="WMI:unspecified"

---splunk-wmi-end-of-event---
20120313033326.660213
AvgDiskQueueLength=0
DiskReadBytesPersec=0
Name=_Total
wmi_type=unspecified

---splunk-wmi-end-of-event---

System utility perfmon.exe indicate that the values for these counters are constantly changing.

How to collect these perf counters?

Thanks.

Re: How to collect perf counters of type average or per second?

malmoore — Tue, 13 Mar 2012 16:32:11 GMT

Hi,

From the looks of things it appears that your splunk-wmi command is not attaching to WMI properly.

At the very least, Splunk needs to run as a user that has access to WMI. It also needs to be correctly configured to connect to WMI.

Review the following documentation at your leisure for specifics about how to attach to WMI sources to get data:

http://docs.splunk.com/Documentation/Splunk/latest/Data/MonitorWMIdata (for information on how to connect to WMI sources to get data from WMI)
http://docs.splunk.com/Documentation/Splunk/latest/Data/Real-timeWindowsperformancemonitoring (to learn how to use Splunk's performance monitoring tools, which use WMI-based inputs in some situations)
http://docs.splunk.com/Documentation/Splunk/latest/Data/ConsiderationsfordecidinghowtomonitorWindowsdata (for general information on monitoring remote Windows data)

Cheers.

Re: How to collect perf counters of type average or per second?

donhuanmatus — Mon, 28 Sep 2020 11:30:48 GMT

Thanks for your reply.

Splunk-WMI command is correctly attached to WMI.

This confirms the successful execution of the following query:

D:\Splunk\bin>splunk-wmi -wql "SELECT Name, PercentDiskTime FROM Win32_PerfFormattedData_PerfDisk_PhysicalDisk WHERE Name = '_Total'"
SPLUNK index= source="WMI:unspecified" sourcetype="WMI:unspecified"

---splunk-wmi-end-of-event---
20120314072433.340459
Name=_Total
PercentDiskTime=28
wmi_type=unspecified

---splunk-wmi-end-of-event---

The problem with collecting perf counters of type average and per second.

Re: How to collect perf counters of type average or per second?

malmoore — Wed, 14 Mar 2012 14:45:42 GMT

Hi,

When collecting performance metrics from a local machine, you should not use splunk-wmi. You should use splunk-perfmon. Splunk-perfmon attaches directly to PDH to poll the performance objects and counters directly.

In any case, whenever you use WQL, your query must be syntactically correct, including case, punctuation and spacing. Failure to maintain a correct WQL query structure will result in no data being returned.

Just to make sure that this was not a bug, I tested your WQL query with WBEMTEST. It returns the same results.

More information on collecting performance metrics is here: http://docs.splunk.com/Documentation/Splunk/latest/Data/Real-timeWindowsperformancemonitoring

Cheers.

Re: How to collect perf counters of type average or per second?

donhuanmatus — Mon, 28 Sep 2020 11:31:12 GMT

Actually I use splunk-wmi only to check WQL query. For the collection of perf counters I use Splunk Light Forwarder, that returns zero for perf counters of type average and per second.

This is my config for SLF.

wmi.conf

[WMI:PhysicalDisk]

disabled = 0

interval = 360

server = localhost

wql = SELECT Name, PercentDiskTime, AvgDiskQueueLength FROM Win32_PerfFormattedData_PerfDisk_PhysicalDisk WHERE Name = '_Total'

I want to once again draw your attention that the data is not properly collected only for perf counters of type average and per second. For incremental perf counters everything is fine.

Thanks.

Re: How to collect perf counters of type average or per second?

mikhail_mikheev — Thu, 15 Mar 2012 08:49:54 GMT

I'd like to add some details. We worked toghether with donhuanmatus on this issue. The real problem is that we have sporadical values for the counters when we collect them by Light Weight Forwarder (LWF). More precisely speaking we have very few values from a bunch of counter of the types he mentioned collected every 6 minutes. And for example we have only 5 or 10 sporadical values for them during the day. We collect our own counters that are 'average value' and 'value per second'. The good example of counters of that types are 'Physical Disk\Avg. Disk Queue Length' and ' Physical Disk\Disk Read Bytes/sec'. We also have a number of simple incremental counters that are successfully collected, so the problem is with the counters of that specific types.

When we started debugging the WQL used by LWF we faced that splunk-wmi returns zero all the time in spite non-zero values are shown in Performance Monitor. So we also tried to collect anything for Windows built-in counters like the ones mentioned above from 'Physical Disk' category and had no success.

malmoore wrote that WBEMTEST also did't return values for counters 'Avg. Disk Queue Length' and ' Physical Disk\Disk Read Bytes/sec', which gets me confused. So let me rephrase the question. Could someone experienced provide an example of WQL that collects counters 'Physical Disk\Avg. Disk Queue Length' and ' Physical Disk\Disk Read Bytes/sec' correctly. Having such example we can dig deper and make our own counters work.

Thanks, any help will be much appreciated!

Re: How to collect perf counters of type average or per second?

malmoore — Fri, 16 Mar 2012 04:49:17 GMT

Can you tell me which version of Splunk you are using? Thanks.

Re: How to collect perf counters of type average or per second?

malmoore — Fri, 16 Mar 2012 06:36:19 GMT

So I just wanted to provide an update to this.

After some internal discussion, I've decided to open a bug on this issue. While we are not convinced that Splunk is generating invalid data, we are investigating whether or not the data being returned by WMI is valid.

We have, so far, concluded unequivocally that splunk-wmi does, in fact, return non-zero data for average- and per-second performance counters gathered against the local WMI namespace. The questions that remain are:

How reliable or valid is this data?
How is splunk-wmi presenting this data?

Once our investigation is complete, I'll post an update here. In the meantime, there are ways to work around this possible issue:

Use a universal forwarder to collect data from your remote Windows machines, instead of a light forwarder.
Use performance monitoring inputs to get your data, instead of WMI.

Cheers.

Re: How to collect perf counters of type average or per second?

malmoore — Tue, 20 Mar 2012 12:42:44 GMT

Finally, another update. Thanks for your patience.

The 'anomaly' you're seeing with splunk-wmi returning 0 values is directly attributed to the fact that the WMI specification defines Win32_PerfFormattedData structures as unsigned integers. Performance Monitor allows for decimal values.

What you're seeing, particularly with Average Disk Queue Length, is WMI rounding the value down to 0. If you compare WMI based data to Performance Monitor-based data, you'll see that the Perfmon data is also very close to 0 (but is not 0).

It's always best to use Performance Monitoring inputs instead of WMI inputs to get the data you need. I've already provided links to the appropriate documentation. We'll update the documentation to include additional information on the discrepancies between WMI and Perfmon inputs.

Cheers!

Re: How to collect perf counters of type average or per second?

mikhail_mikheev — Tue, 20 Mar 2012 13:16:05 GMT

Thanks a lot! We will dig into this direction.

Re: How to collect perf counters of type average or per second?

donhuanmatus — Mon, 28 Sep 2020 11:36:12 GMT

One question, please.

When we use WIMI we could specify the instances of the perf. counter as follows:

[WMI:RedisService_Private_Bytes]

disabled = 0

interval = 360

server = localhost

wql = SELECT Name, PrivateBytes FROM Win32_PerfFormattedData_PerfProc_Process WHERE LIKE 'redis-server%'

Performance monitoring inputs allows you to do something like that, or need to explicitly specify the name?
In the documentation I could not find something like that.

Now we make the following way, but it's not very convenient:

[SystemPerfmon:Process]

interval = 360

object = Process

counters = Private Bytes

instances = redis-server0;redis-server1;redis-server2;redis-server3;redis-server4;redis-server5;redis-server6;redis-server7;redis-server8

disabled = 0

index = sns

Thanks.