All Apps and Add-ons

Why are counters from Perfmon not being extracted?

lwass
Explorer

Hello,

I have the following in my inputs.conf on a Windows server:

[perfmon://CPU]
counters = % Processor Time; % User Time; % Privileged Time; Interrupts/sec; % DPC Time; % Interrupt Time; DPCs Queued/sec; DPC Rate; % Idle Time; % C1 Time; % C2 Time; % C3 Time; C1 Transitions/sec; C2 Transitions/sec; C3 Transitions/sec
disabled = 0
instances = *
interval = 10
object = Processor
useEnglishOnly=true
index = os

I can see metrics coming through:

0   7.597075517979601   3.4455327772956763  4.071993282258527   338.6526155305428   0   0   55.64008336869486   0   91.11271353582889   2.18562547981863    88.92708805601026   0   26.86764386091932   250.12974415296156  0   

With object=Processor, sourcetype=PerfmonMk:CPU.

The Windows Infrastructure app requires a counter field be present in its searches, but Splunk does not appear to be including this field in the results.

Has anyone seen this before? Do you know where the extraction may be failing?

0 Karma

marianomromano
Engager

For those of you that did not understand what mode = single means, below is an example of the setting that needs to be changed. I was one of those people that did not understand 😉

[perfmon://CPU]
counters = % Processor Time; % User Time; % Privileged Time; % Idle Time
disabled = 0
instances = *
interval = 10
mode = single (This use to be mode=multikv)
object = Processor
useEnglishOnly=true
index = windows

0 Karma

casccep
Engager

The problem is the Splunk App for Windows Infrastructure, even on version 1.5.2 does not fully supports the new standards on the Splunk Add-on for Microsoft Windows.

It basically have 2 problems :

1) You can't use XML (which is the default in the TA v6.0)
2) You can't use multikv (which is also the default in the TA v6.0)

So, you need to disable XML (renderXml = false) in all your windows event inputs, as well as disable ** multikv (mode = single)** in the performance ones.

With default configurations, single mode in performance can increase indexed data (so licence use) by almost 5x so, be carefull.

Other option... is you can modify the app, so it takes the data correctly with the new format.
For the performance ones for example is easy to modify... the problem is that there are searches that looks for a "Counter" that does not exist in multikv mode ... but you can fix this just by manually put the values for performance, like :
In the file: /opt/splunk/etc/apps/splunk_app_windows_infrastructure/local/data/ui/views/windows_performance.xml

Search for CPUCounter token, and change the input to :

  <input type="dropdown" token="CPUCounter" searchWhenChanged="true">
    <label>Counter</label>
    <initialValue>%_Processor_Time</initialValue>
    <choice value="%_Processor_Time">% Processor Time</choice>
    <choice value="%_User_Time">% User Time</choice>
    <choice value="%_Privileged_Time">% Privileged Time</choice>
    <choice value="Interrupts/sec">Interrupts/sec</choice>
    <choice value="%_DPC_Time">% DPC Time</choice>
    <choice value="%_Interrupt_Time">% Interrupt Time</choice>
  </input>

This is a reduced example, if you want all the counters, look at your inputs.conf you will have all them in each input, the secret is that you need to put the "_" underscore replacing spaces in the value to make it work, and add the "choice" for this counter.
You can ofcourse make a scheduled search that makes a CSV automatically and then you get the values from there... but I feel it easier this way as it will not consume search.

I am attaching a modified (not all options, just the ones we use now), so you may want to add all the choices, but it works with multikv.
OOPS... I can't attach a file... I need more KARMA to attach files! ... if you provide it, I will attach the file(s) I have.

kainitz64
Explorer

@casccep 

great and useful , thanks a lot....

article is from 2019 but still valid and helpful.
if you have enough "KARMA" now maybe you can share the file you've mentioned in the article

thanks a lot

happy splunking

0 Karma

bhargavnariyani
Path Finder

Please add mode = single in your input stanza.
The data is currently ingested in multikv mode. Adding the above parameter would give you the perfmon data in single mode which can be used by Windows Infrastructure app.

whrg
Motivator

I believe you must install the Splunk Add-on for Microsoft Windows:

https://splunkbase.splunk.com/app/742/

The add-on comes with lots of field extractions. It also includes an inputs.conf which should be similar to yours.

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...