Getting Data In

Windows Perfmon data not collecting

mooree
Path Finder

Splunk is faliing to collect perfmon data from our Windows 2022 servers. 

I've extracted and deployed the stanzas from the Splunk TA for windows to collect selected perfmon stats from servers. We use a deployment server to push this out. Here's a sample:

 

 

[perfmon://CPU]
counters = % Processor Time 
disabled = 0
instances = *
interval = 10
mode = single
object = Processor
useEnglishOnly=true
index=2_###_test

 

 

The Splunk Universal Forwarder now restarts as expected on deployment (missed that first time 😉) .  There are no apparent errors in splunkd.log. 

Nothing turns up! Metrics confirms nothing being sent to that index from the UF. 

I'm guessing that our Security lockdown is preventing collection, but with no error messages anywhere it's hard to diagnose! 

Perfmon works on the server target so we know that the data is there and working. 

Splunk is 9.2.1. it's running in "least privilege" mode on the UF (the new default). 

Any hints and pointers most welcome!  

Labels (1)
0 Karma

psla
Explorer

Hi All

Has anyone managed to solve this issue without reinstalling UF?

We have this problem only on certain Window Servers 2022. Other windows versions are not affected. Also not all Win2022 are affected, only certain machines

Command "Get-counter -ListSet *" returns the following error.

Could not find any performance counter sets on the computer: error c0000bc8. Verify that the computer exists, that it is discoverable, and that you have sufficient privileges to view performance counter data on that computer

Perfmon counters are available for other users on this machine, so there is problem for SplunkForwarder user. 

I've used the "lodctr /R" command but issue still persists. The issue occurred immediately after the upgrade to version 9.1.5, so it's definitely Splunk problem

0 Karma

PickleRick
SplunkTrust
SplunkTrust
0 Karma

inventsekar
SplunkTrust
SplunkTrust

at times these simple issues may give us big headache. 

the shortest troubleshooting step is to resinstall the agent.. (do this only if you have min custom configs in the UF)

thanks and best regards,
Sekar

PS - If this or any post helped you in any way, pls consider upvoting, thanks for reading !
0 Karma

JohnEGones
Communicator

This may be a relevant source for additional troubleshooting:

Solved: What's the best way to get Windows Perfmon data in... - Splunk Community

0 Karma

JohnEGones
Communicator

@mooree 

You write:

"

  • All other logs and events are getting through fine.  "

    these are  (other  - non-metric) logs from that 2022 server?
0 Karma

mooree
Path Finder

Yes - It's only perfmon data we're not getting. Splunk internals and event log events are both OK. AFAIK (and intended) these are not being collected as metrics. 

I'd been through the article you referenced, and heve now been back and checked my workings.  We've not installed the Windows add-on to every layer yet - I've just used bit of inputs.conf from it initially to get the data to look at and will then go back to all the clever bit once the basics are working. 

0 Karma

JohnEGones
Communicator

Per the DOCS, here: Install the Splunk Add-on for Windows - Splunk Documentation

and for metric here: https://docs.splunk.com/Documentation/AddOns/released/Windows/Configuration#Collect_perfmon_data_and...

You should ensure you have a metrics index defined, and install it accordingly at every layer to ensure you're getting the data you need. 

0 Karma

JohnEGones
Communicator

What do you mean by "Security Lockdown"? Are there any local host firewall settings that are active on that server?

0 Karma

mooree
Path Finder

We apply a range of GPO settings to get us close to a CIS Level One hardening. This does usually include the Windows Firewall, but it's set to off where it needs to be and it's off here. 

0 Karma

inventsekar
SplunkTrust
SplunkTrust

Hi @mooree 

from the UF, do you receive other regular logs/app logs to the indexer?

using the btool, pls verify if the perfmon input is getting read by UF.. 

$SPLUNK_HOME$/bin/splunk btool inputs list --debug

 

thanks and best regards,
Sekar

PS - If this or any post helped you in any way, pls consider upvoting, thanks for reading !
0 Karma

mooree
Path Finder

Thanks for the thoughts - I've re-checked both and:

  • inputs all good and showing  in the btool output.
  • All other logs and events are getting through fine.  

 

0 Karma
Get Updates on the Splunk Community!

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...