Windows Perfmon data not collecting

[mooree]

Splunk is faliing to collect perfmon data from our Windows 2022 servers. 

I've extracted and deployed the stanzas from the Splunk TA for windows to collect selected perfmon stats from servers. We use a deployment server to push this out. Here's a sample:

[perfmon://CPU]
counters = % Processor Time 
disabled = 0
instances = *
interval = 10
mode = single
object = Processor
useEnglishOnly=true
index=2_###_test

The Splunk Universal Forwarder now restarts as expected on deployment (missed that first time 😉) .  There are no apparent errors in splunkd.log. 

Nothing turns up! Metrics confirms nothing being sent to that index from the UF. 

I'm guessing that our Security lockdown is preventing collection, but with no error messages anywhere it's hard to diagnose! 

Perfmon works on the server target so we know that the data is there and working. 

Splunk is 9.2.1. it's running in "least privilege" mode on the UF (the new default). 

Any hints and pointers most welcome!  

[psla]

Hi All

Has anyone managed to solve this issue without reinstalling UF?

We have this problem only on certain Window Servers 2022. Other windows versions are not affected. Also not all Win2022 are affected, only certain machines

Command "Get-counter -ListSet *" returns the following error.

Could not find any performance counter sets on the computer: error c0000bc8. Verify that the computer exists, that it is discoverable, and that you have sufficient privileges to view performance counter data on that computer

Perfmon counters are available for other users on this machine, so there is problem for SplunkForwarder user. 

I've used the "lodctr /R" command but issue still persists. The issue occurred immediately after the upgrade to version 9.1.5, so it's definitely Splunk problem

[PickleRick]

See this thread https://community.splunk.com/t5/Getting-Data-In/Debugging-perfmon-input/m-p/621539#M107042

[inventsekar]

at times these simple issues may give us big headache. 

the shortest troubleshooting step is to resinstall the agent.. (do this only if you have min custom configs in the UF)

[JohnEGones]

This may be a relevant source for additional troubleshooting:

Solved: What's the best way to get Windows Perfmon data in... - Splunk Community

[JohnEGones]

@mooree 

You write:

"

• All other logs and events are getting through fine.  "
  
  these are  (other  - non-metric) logs from that 2022 server?

[mooree]

Yes - It's only perfmon data we're not getting. Splunk internals and event log events are both OK. AFAIK (and intended) these are not being collected as metrics. 

I'd been through the article you referenced, and heve now been back and checked my workings.  We've not installed the Windows add-on to every layer yet - I've just used bit of inputs.conf from it initially to get the data to look at and will then go back to all the clever bit once the basics are working. 

[JohnEGones]

Per the DOCS, here: Install the Splunk Add-on for Windows - Splunk Documentation

and for metric here: https://docs.splunk.com/Documentation/AddOns/released/Windows/Configuration#Collect_perfmon_data_and...

You should ensure you have a metrics index defined, and install it accordingly at every layer to ensure you're getting the data you need. 

[JohnEGones]

What do you mean by "Security Lockdown"? Are there any local host firewall settings that are active on that server?

[mooree]

We apply a range of GPO settings to get us close to a CIS Level One hardening. This does usually include the Windows Firewall, but it's set to off where it needs to be and it's off here. 

[inventsekar]

Hi @mooree 

from the UF, do you receive other regular logs/app logs to the indexer?

using the btool, pls verify if the perfmon input is getting read by UF.. 

$SPLUNK_HOME$/bin/splunk btool inputs list --debug

[mooree]

Thanks for the thoughts - I've re-checked both and:

• inputs all good and showing  in the btool output.
• All other logs and events are getting through fine.