Debugging perfmon input?

[PickleRick]

Hello there.

I tried to set up perfmon inputs to capture state of my windows 10 test box.

Aaaaand. It's not working. And I have no idea how I can debug it further.

The inputs seem to be defined properly (I don't understand why there are two identical definitions for perfmon://CPU and perfmon://Processor but while testing I tried running with just one perfmon input enabled and the result was the same so it's definitely not the result of overlapping inputs).

PS C:\Program Files\SplunkUniversalForwarder\bin> .\splunk.exe btool inputs list perfmon://CPU
[perfmon://CPU]
counters = % Processor Time; % User Time; % Privileged Time; Interrupts/sec; % DPC Time; % Interrupt Time; DPCs Queued/sec; DPC Rate; % Idle Time; % C1 Time; % C2 Time; % C3 Time; C1 Transitions/sec; C2 Transitions/sec; C3 Transitions/sec
disabled = 0
host = dziura
index = winmetrics
instances = *
interval = 300
mode = multikv
object = Processor
useEnglishOnly = true
PS C:\Program Files\SplunkUniversalForwarder\bin> .\splunk.exe btool inputs list perfmon://Process
[perfmon://Process]
counters = % Processor Time; % User Time; % Privileged Time; Virtual Bytes Peak; Virtual Bytes; Page Faults/sec; Working Set Peak; Working Set; Page File Bytes Peak; Page File Bytes; Private Bytes; Thread Count; Priority Base; Elapsed Time; ID Process; Creating Process ID; Pool Paged Bytes; Pool Nonpaged Bytes; Handle Count; IO Read Operations/sec; IO Write Operations/sec; IO Data Operations/sec; IO Other Operations/sec; IO Read Bytes/sec; IO Write Bytes/sec; IO Data Bytes/sec; IO Other Bytes/sec; Working Set - Private
disabled = 0
host = dziura
index = winmetrics
instances = *
interval = 300
mode = multikv
object = Process
useEnglishOnly = true
[perfmon://Processor]
counters = % Processor Time; % User Time; % Privileged Time; Interrupts/sec; % DPC Time; % Interrupt Time; DPCs Queued/sec; DPC Rate; % Idle Time; % C1 Time; % C2 Time; % C3 Time; C1 Transitions/sec; C2 Transitions/sec; C3 Transitions/sec
disabled = 0
host = dziura
index = winmetrics
instances = *
interval = 300
mode = multikv
object = Processor
useEnglishOnly = true

The list inputstatus shows:

C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe
exit status description = exited with code -1
time closed = 2022-11-21T09:27:17+0100
time opened = 2022-11-21T09:27:14+0100

I raised logging level for modularinputs and execprocessor to DEBUG but still it's not helpful:

11-21-2022 09:27:10.491 +0100 DEBUG ModularInputs [6028 MainThread] - Found scheme="perfmon".
11-21-2022 09:27:10.491 +0100 DEBUG ModularInputs [6028 MainThread] - Locating script for scheme="perfmon"...
11-21-2022 09:27:10.491 +0100 DEBUG ModularInputs [6028 MainThread] - No regular file="C:\Program Files\SplunkUniversalForwarder\etc\system\windows_x86_64\bin\perfmon.bat".
11-21-2022 09:27:10.492 +0100 DEBUG ModularInputs [6028 MainThread] - No regular file="C:\Program Files\SplunkUniversalForwarder\etc\system\windows_x86_64\bin\perfmon.cmd".
11-21-2022 09:27:10.492 +0100 DEBUG ModularInputs [6028 MainThread] - No regular file="C:\Program Files\SplunkUniversalForwarder\etc\system\windows_x86_64\bin\perfmon.py".
11-21-2022 09:27:10.492 +0100 DEBUG ModularInputs [6028 MainThread] - No regular file="C:\Program Files\SplunkUniversalForwarder\etc\system\windows_x86_64\bin\perfmon.js".
11-21-2022 09:27:10.492 +0100 DEBUG ModularInputs [6028 MainThread] - No regular file="C:\Program Files\SplunkUniversalForwarder\etc\system\windows_x86_64\bin\perfmon.exe".
11-21-2022 09:27:10.492 +0100 DEBUG ModularInputs [6028 MainThread] - No regular file="C:\Program Files\SplunkUniversalForwarder\etc\system\windows_x86_64\bin\perfmon.bat".
11-21-2022 09:27:10.492 +0100 DEBUG ModularInputs [6028 MainThread] - No regular file="C:\Program Files\SplunkUniversalForwarder\etc\system\windows_x86_64\bin\perfmon.cmd".
11-21-2022 09:27:10.492 +0100 DEBUG ModularInputs [6028 MainThread] - No regular file="C:\Program Files\SplunkUniversalForwarder\etc\system\windows_x86_64\bin\perfmon.py".
11-21-2022 09:27:10.492 +0100 DEBUG ModularInputs [6028 MainThread] - No regular file="C:\Program Files\SplunkUniversalForwarder\etc\system\windows_x86_64\bin\perfmon.js".
11-21-2022 09:27:10.492 +0100 DEBUG ModularInputs [6028 MainThread] - No regular file="C:\Program Files\SplunkUniversalForwarder\etc\system\windows_x86_64\bin\perfmon.exe".
11-21-2022 09:27:10.493 +0100 DEBUG ModularInputs [6028 MainThread] - No regular file="C:\Program Files\SplunkUniversalForwarder\etc\system\bin\perfmon.bat".
11-21-2022 09:27:10.493 +0100 DEBUG ModularInputs [6028 MainThread] - Found script ""C:\Program Files\SplunkUniversalForwarder\etc\system\bin\perfmon.cmd"" to handle scheme "perfmon".
11-21-2022 09:27:10.614 +0100 DEBUG ModularInputs [6028 MainThread] - Introspecting scheme=perfmon: exited: status=done, exit=0
11-21-2022 09:27:10.614 +0100 DEBUG ModularInputs [6028 MainThread] - XML scheme path "\scheme\script": "script" -> "splunk-perfmon.path"
11-21-2022 09:27:10.614 +0100 DEBUG ModularInputs [6028 MainThread] - XML endpoint path "\scheme\endpoint\id": "id" -> "win-perfmon"
11-21-2022 09:27:10.614 +0100 DEBUG ModularInputs [6028 MainThread] - Setting up values from introspection for scheme "perfmon".
11-21-2022 09:27:10.614 +0100 DEBUG ModularInputs [6028 MainThread] - Locating script for scheme="perfmon"...
11-21-2022 09:27:10.614 +0100 DEBUG ModularInputs [6028 MainThread] - No regular file="C:\Program Files\SplunkUniversalForwarder\etc\system\windows_x86_64\bin\splunk-perfmon.path".
11-21-2022 09:27:10.614 +0100 DEBUG ModularInputs [6028 MainThread] - No regular file="C:\Program Files\SplunkUniversalForwarder\etc\system\windows_x86_64\bin\splunk-perfmon.path.exe".
11-21-2022 09:27:10.614 +0100 DEBUG ModularInputs [6028 MainThread] - No regular file="C:\Program Files\SplunkUniversalForwarder\etc\system\windows_x86_64\bin\splunk-perfmon.path".
11-21-2022 09:27:10.614 +0100 DEBUG ModularInputs [6028 MainThread] - No regular file="C:\Program Files\SplunkUniversalForwarder\etc\system\windows_x86_64\bin\splunk-perfmon.path.exe".
11-21-2022 09:27:10.615 +0100 DEBUG ModularInputs [6028 MainThread] - No regular file="C:\Program Files\SplunkUniversalForwarder\etc\system\bin\splunk-perfmon.path".
11-21-2022 09:27:10.615 +0100 DEBUG ModularInputs [6028 MainThread] - No regular file="C:\Program Files\SplunkUniversalForwarder\etc\system\bin\splunk-perfmon.path.exe".
11-21-2022 09:27:10.615 +0100 DEBUG ModularInputs [6028 MainThread] - No regular file="C:\Program Files\SplunkUniversalForwarder\etc\system\bin\splunk-perfmon.path".
11-21-2022 09:27:10.615 +0100 DEBUG ModularInputs [6028 MainThread] - No regular file="C:\Program Files\SplunkUniversalForwarder\etc\system\bin\splunk-perfmon.path.exe".
11-21-2022 09:27:10.615 +0100 DEBUG ModularInputs [6028 MainThread] - Found script ""C:\Program Files\SplunkUniversalForwarder\bin\scripts\splunk-perfmon.path"" to handle scheme "perfmon".
11-21-2022 09:27:10.615 +0100 DEBUG ModularInputs [6028 MainThread] - For scheme "perfmon" found script "splunk-perfmon.path" at path ""C:\Program Files\SplunkUniversalForwarder\bin\scripts\splunk-perfmon.path""
11-21-2022 09:27:10.615 +0100 DEBUG ModularInputs [6028 MainThread] - Setting "id" to "win-perfmon".
11-21-2022 09:27:11.098 +0100 DEBUG ExecProcessor [12380 ExecProcessor] - In configure(), looking at stanza: [script://C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe] -> {host -> dziura, source -> perfmon, sourcetype -> perfmon}
11-21-2022 09:27:11.098 +0100 DEBUG ExecProcessor [12380 ExecProcessor] - Stanza='script://C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe' isModInput=true isIntrospectionInput=false
11-21-2022 09:27:11.098 +0100 DEBUG ExecProcessor [12380 ExecProcessor] - getInterpreterPathFor(): scriptPath=C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe pyVersStr=
11-21-2022 09:27:11.098 +0100 DEBUG ExecProcessor [12380 ExecProcessor] - After normalization script is ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe""
11-21-2022 09:27:11.098 +0100 DEBUG ExecProcessor [12380 ExecProcessor] - stanza=script://C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe interval=18446744073709551.615
11-21-2022 09:27:11.098 +0100 DEBUG ExecProcessor [12380 ExecProcessor] - Creating an ExecedCommand, cmd='"C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe"', args={"C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe"}, runViaShell=false
11-21-2022 09:27:11.098 +0100 DEBUG ExecProcessor [12380 ExecProcessor] - ExecProcessorSharedState::addToRunQueue() path='"C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe"' restartTimerIfNeeded=0
11-21-2022 09:27:11.098 +0100 DEBUG ExecProcessor [12380 ExecProcessor] - adding ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe"" to runqueue
11-21-2022 09:27:11.098 +0100 DEBUG ExecProcessor [12380 ExecProcessor] - cmd='"C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe"' Added to run queue
11-21-2022 09:27:11.098 +0100 DEBUG ExecProcessor [12380 ExecProcessor] - Creating InputStatusHandler for group="modular input commands" key="C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe"
11-21-2022 09:27:11.098 +0100 DEBUG ExecProcessor [12380 ExecProcessor] - Done configuring ExecedCommand: command='"C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe"' runViaShell=0 tickStarted=0 running=0 state=WAITING_ON_RUNQUEUE interval=18446744073709551.615
11-21-2022 09:27:14.883 +0100 DEBUG ExecProcessor [12380 ExecProcessor] - Running: "C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" on PipelineSet 0
11-21-2022 09:27:14.883 +0100 DEBUG ExecProcessor [12380 ExecProcessor] - PipelineSet 0: Created new ExecedCommandPipe for ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe"", uniqueId=5
11-21-2022 09:27:16.532 +0100 DEBUG ExecProcessor [12380 ExecProcessor] - PipelineSet 0: Got EOF from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe"", uniqueId=5
11-21-2022 09:27:17.048 +0100 DEBUG ExecProcessor [12380 ExecProcessor] - PipelineSet 0: Ran script: "C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe", took 2.172 seconds to run, 0 bytes read 0 events read, status=done, exit=4294967295
11-21-2022 09:27:17.048 +0100 DEBUG ExecProcessor [12380 ExecProcessor] - PipelineSet 0: Destroying ExecedCommandPipe for ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe"" id=5
11-21-2022 09:27:17.048 +0100 DEBUG ExecProcessor [12380 ExecProcessor] - cmd='"C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe"' Not added to run queue

 The only relevant entry here is the line with "exit=4294967295" which corresponds to the inputstatus message that the process exited with -1. But I still don't know why.

I accept that the reason may be completely on the windows side but I would like to be able to diagnose why.

Oh, and yes - I did try the lodctr.exe /r - nothing changes.

The UF is running as LOCAL SYSTEM so it should not have permission issues.

Also I can run perfmon.msc and it's showing the counters properly.

Any more debug ideas? I'm stuck.

[PickleRick]

Just to let you know - after some debugging (thx j.ho!) it turns out that my windows is "multilingual" somehow (even though I'm using an english GUI) and UF gets from the system non-english counter names and can't subscribe to them when useEnglishOnly = true. If I set it to false (and give localized object and counter names), UF is able to pull the metrics.

To make things even more puzzling - windows' own perfmon.msc shows the objects and counters with english names.