Getting Data In

How to collect perf counters of type average or per second?

donhuanmatus
Explorer

Hi Everybody,

I have a WMI Perf counter query that always returns zero in splunk-wmi.exe for counters with the following types: average and per second.

I run the folowing command:

D:\Splunk\bin>splunk-wmi -wql "SELECT Name, DiskReadBytesPerSec, AvgDiskQueueLength FROM Win32_PerfFormattedData_PerfDisk_PhysicalDisk"

And get the following output:

SPLUNK index= source="WMI:unspecified" sourcetype="WMI:unspecified"

---splunk-wmi-end-of-event---
20120313033326.660213
AvgDiskQueueLength=0
DiskReadBytesPersec=0
Name=0 C: 😧
wmi_type=unspecified

---splunk-wmi-end-of-event---
SPLUNK index= source="WMI:unspecified" sourcetype="WMI:unspecified"

---splunk-wmi-end-of-event---
20120313033326.660213
AvgDiskQueueLength=0
DiskReadBytesPersec=0
Name=_Total
wmi_type=unspecified

---splunk-wmi-end-of-event---

System utility perfmon.exe indicate that the values for these counters are constantly changing.

How to collect these perf counters?

Thanks.

Tags (1)
1 Solution

malmoore
Splunk Employee
Splunk Employee

Finally, another update. Thanks for your patience.

The 'anomaly' you're seeing with splunk-wmi returning 0 values is directly attributed to the fact that the WMI specification defines Win32_PerfFormattedData structures as unsigned integers. Performance Monitor allows for decimal values.

What you're seeing, particularly with Average Disk Queue Length, is WMI rounding the value down to 0. If you compare WMI based data to Performance Monitor-based data, you'll see that the Perfmon data is also very close to 0 (but is not 0).

It's always best to use Performance Monitoring inputs instead of WMI inputs to get the data you need. I've already provided links to the appropriate documentation. We'll update the documentation to include additional information on the discrepancies between WMI and Perfmon inputs.

Cheers!

View solution in original post

donhuanmatus
Explorer

One question, please.

When we use WIMI we could specify the instances of the perf. counter as follows:

[WMI:RedisService_Private_Bytes]

disabled = 0

interval = 360

server = localhost

wql = SELECT Name, PrivateBytes FROM Win32_PerfFormattedData_PerfProc_Process WHERE LIKE 'redis-server%'

Performance monitoring inputs allows you to do something like that, or need to explicitly specify the name?
In the documentation I could not find something like that.

Now we make the following way, but it's not very convenient:

[SystemPerfmon:Process]

interval = 360

object = Process

counters = Private Bytes

instances = redis-server0;redis-server1;redis-server2;redis-server3;redis-server4;redis-server5;redis-server6;redis-server7;redis-server8

disabled = 0

index = sns

Thanks.

mikhail_mikheev
Engager

Thanks a lot! We will dig into this direction.

malmoore
Splunk Employee
Splunk Employee

Finally, another update. Thanks for your patience.

The 'anomaly' you're seeing with splunk-wmi returning 0 values is directly attributed to the fact that the WMI specification defines Win32_PerfFormattedData structures as unsigned integers. Performance Monitor allows for decimal values.

What you're seeing, particularly with Average Disk Queue Length, is WMI rounding the value down to 0. If you compare WMI based data to Performance Monitor-based data, you'll see that the Perfmon data is also very close to 0 (but is not 0).

It's always best to use Performance Monitoring inputs instead of WMI inputs to get the data you need. I've already provided links to the appropriate documentation. We'll update the documentation to include additional information on the discrepancies between WMI and Perfmon inputs.

Cheers!

malmoore
Splunk Employee
Splunk Employee

So I just wanted to provide an update to this.

After some internal discussion, I've decided to open a bug on this issue. While we are not convinced that Splunk is generating invalid data, we are investigating whether or not the data being returned by WMI is valid.

We have, so far, concluded unequivocally that splunk-wmi does, in fact, return non-zero data for average- and per-second performance counters gathered against the local WMI namespace. The questions that remain are:

  • How reliable or valid is this data?
  • How is splunk-wmi presenting this data?

Once our investigation is complete, I'll post an update here. In the meantime, there are ways to work around this possible issue:

Cheers.

mikhail_mikheev
Engager

I'd like to add some details. We worked toghether with donhuanmatus on this issue. The real problem is that we have sporadical values for the counters when we collect them by Light Weight Forwarder (LWF). More precisely speaking we have very few values from a bunch of counter of the types he mentioned collected every 6 minutes. And for example we have only 5 or 10 sporadical values for them during the day. We collect our own counters that are 'average value' and 'value per second'. The good example of counters of that types are 'Physical Disk\Avg. Disk Queue Length' and ' Physical Disk\Disk Read Bytes/sec'. We also have a number of simple incremental counters that are successfully collected, so the problem is with the counters of that specific types.

When we started debugging the WQL used by LWF we faced that splunk-wmi returns zero all the time in spite non-zero values are shown in Performance Monitor. So we also tried to collect anything for Windows built-in counters like the ones mentioned above from 'Physical Disk' category and had no success.

malmoore wrote that WBEMTEST also did't return values for counters 'Avg. Disk Queue Length' and ' Physical Disk\Disk Read Bytes/sec', which gets me confused. So let me rephrase the question. Could someone experienced provide an example of WQL that collects counters 'Physical Disk\Avg. Disk Queue Length' and ' Physical Disk\Disk Read Bytes/sec' correctly. Having such example we can dig deper and make our own counters work.

Thanks, any help will be much appreciated!

donhuanmatus
Explorer

Actually I use splunk-wmi only to check WQL query. For the collection of perf counters I use Splunk Light Forwarder, that returns zero for perf counters of type average and per second.

This is my config for SLF.

wmi.conf

[WMI:PhysicalDisk]

disabled = 0

interval = 360

server = localhost

wql = SELECT Name, PercentDiskTime, AvgDiskQueueLength FROM Win32_PerfFormattedData_PerfDisk_PhysicalDisk WHERE Name = '_Total'

I want to once again draw your attention that the data is not properly collected only for perf counters of type average and per second. For incremental perf counters everything is fine.

Thanks.

0 Karma

malmoore
Splunk Employee
Splunk Employee

Can you tell me which version of Splunk you are using? Thanks.

0 Karma

malmoore
Splunk Employee
Splunk Employee

Hi,

When collecting performance metrics from a local machine, you should not use splunk-wmi. You should use splunk-perfmon. Splunk-perfmon attaches directly to PDH to poll the performance objects and counters directly.

In any case, whenever you use WQL, your query must be syntactically correct, including case, punctuation and spacing. Failure to maintain a correct WQL query structure will result in no data being returned.

Just to make sure that this was not a bug, I tested your WQL query with WBEMTEST. It returns the same results.

More information on collecting performance metrics is here: http://docs.splunk.com/Documentation/Splunk/latest/Data/Real-timeWindowsperformancemonitoring

Cheers.

0 Karma

donhuanmatus
Explorer

Thanks for your reply.

Splunk-WMI command is correctly attached to WMI.

This confirms the successful execution of the following query:

D:\Splunk\bin>splunk-wmi -wql "SELECT Name, PercentDiskTime FROM Win32_PerfFormattedData_PerfDisk_PhysicalDisk WHERE Name = '_Total'"
SPLUNK index= source="WMI:unspecified" sourcetype="WMI:unspecified"

---splunk-wmi-end-of-event---
20120314072433.340459
Name=_Total
PercentDiskTime=28
wmi_type=unspecified

---splunk-wmi-end-of-event---

The problem with collecting perf counters of type average and per second.

0 Karma

malmoore
Splunk Employee
Splunk Employee

Hi,

From the looks of things it appears that your splunk-wmi command is not attaching to WMI properly.

At the very least, Splunk needs to run as a user that has access to WMI. It also needs to be correctly configured to connect to WMI.

Review the following documentation at your leisure for specifics about how to attach to WMI sources to get data:

http://docs.splunk.com/Documentation/Splunk/latest/Data/MonitorWMIdata (for information on how to connect to WMI sources to get data from WMI)
http://docs.splunk.com/Documentation/Splunk/latest/Data/Real-timeWindowsperformancemonitoring (to learn how to use Splunk's performance monitoring tools, which use WMI-based inputs in some situations)
http://docs.splunk.com/Documentation/Splunk/latest/Data/ConsiderationsfordecidinghowtomonitorWindows... (for general information on monitoring remote Windows data)

Cheers.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...