topic Re: Windows Perfmon data not collecting in Getting Data In

Windows Perfmon data not collecting

mooree — Wed, 10 Jul 2024 08:41:40 GMT

Splunk is faliing to collect perfmon data from our Windows 2022 servers.

I've extracted and deployed the stanzas from the Splunk TA for windows to collect selected perfmon stats from servers. We use a deployment server to push this out. Here's a sample:

[perfmon://CPU] counters = % Processor Time disabled = 0 instances = * interval = 10 mode = single object = Processor useEnglishOnly=true index=2_###_test

The Splunk Universal Forwarder now restarts as expected on deployment (missed that first time 😉) . There are no apparent errors in splunkd.log.

Nothing turns up! Metrics confirms nothing being sent to that index from the UF.

I'm guessing that our Security lockdown is preventing collection, but with no error messages anywhere it's hard to diagnose!

Perfmon works on the server target so we know that the data is there and working.

Splunk is 9.2.1. it's running in "least privilege" mode on the UF (the new default).

Any hints and pointers most welcome!

Re: Windows Perfmon data not collecting

inventsekar — Wed, 10 Jul 2024 10:22:26 GMT

Hi @mooree

from the UF, do you receive other regular logs/app logs to the indexer?

using the btool, pls verify if the perfmon input is getting read by UF..

$SPLUNK_HOME$/bin/splunk btool inputs list --debug

Re: Windows Perfmon data not collecting

JohnEGones — Wed, 10 Jul 2024 12:16:48 GMT

What do you mean by "Security Lockdown"? Are there any local host firewall settings that are active on that server?

Re: Windows Perfmon data not collecting

mooree — Wed, 10 Jul 2024 16:00:11 GMT

Thanks for the thoughts - I've re-checked both and:

inputs all good and showing in the btool output.
All other logs and events are getting through fine.

Re: Windows Perfmon data not collecting

mooree — Wed, 10 Jul 2024 16:02:32 GMT

We apply a range of GPO settings to get us close to a CIS Level One hardening. This does usually include the Windows Firewall, but it's set to off where it needs to be and it's off here.

Re: Windows Perfmon data not collecting

JohnEGones — Wed, 10 Jul 2024 16:05:54 GMT

@mooree

You write:

"

All other logs and events are getting through fine. "

these are (other - non-metric) logs from that 2022 server?

Re: Windows Perfmon data not collecting

JohnEGones — Wed, 10 Jul 2024 16:10:10 GMT

This may be a relevant source for additional troubleshooting:

Solved: What's the best way to get Windows Perfmon data in... - Splunk Community

Re: Windows Perfmon data not collecting

mooree — Wed, 10 Jul 2024 16:20:08 GMT

Yes - It's only perfmon data we're not getting. Splunk internals and event log events are both OK. AFAIK (and intended) these are not being collected as metrics.

I'd been through the article you referenced, and heve now been back and checked my workings. We've not installed the Windows add-on to every layer yet - I've just used bit of inputs.conf from it initially to get the data to look at and will then go back to all the clever bit once the basics are working.

Re: Windows Perfmon data not collecting

JohnEGones — Wed, 10 Jul 2024 16:32:18 GMT

Per the DOCS, here: Install the Splunk Add-on for Windows - Splunk Documentation

and for metric here: https://docs.splunk.com/Documentation/AddOns/released/Windows/Configuration#Collect_perfmon_data_and_wmi:uptime_data_in_metric_index

You should ensure you have a metrics index defined, and install it accordingly at every layer to ensure you're getting the data you need.

Re: Windows Perfmon data not collecting

inventsekar — Wed, 10 Jul 2024 21:25:49 GMT

at times these simple issues may give us big headache.

the shortest troubleshooting step is to resinstall the agent.. (do this only if you have min custom configs in the UF)

Re: Windows Perfmon data not collecting

PickleRick — Wed, 10 Jul 2024 21:43:12 GMT

See this thread https://community.splunk.com/t5/Getting-Data-In/Debugging-perfmon-input/m-p/621539#M107042

Re: Windows Perfmon data not collecting

psla — Fri, 02 Aug 2024 13:40:08 GMT

Hi All

Has anyone managed to solve this issue without reinstalling UF?

We have this problem only on certain Window Servers 2022. Other windows versions are not affected. Also not all Win2022 are affected, only certain machines

Command "Get-counter -ListSet *" returns the following error.

Could not find any performance counter sets on the computer: error c0000bc8. Verify that the computer exists, that it is discoverable, and that you have sufficient privileges to view performance counter data on that computer

Perfmon counters are available for other users on this machine, so there is problem for SplunkForwarder user.

I've used the "lodctr /R" command but issue still persists. The issue occurred immediately after the upgrade to version 9.1.5, so it's definitely Splunk problem