Perfmon counters not coming in after Universal Forwarder update to 9.2.6.0 from 9.2.0.1

[DarthHerm]

Thought I would post here in the community as well since I have this opened with support. A couple weeks ago, another agency pushed updates to Splunk Universal Forwarder to half of my hosts without my knowledge or consent. Those hosts were updated to 9.2.6.0 from 9.2.0.1. The updates went unnoticed for a couple weeks since the events from our custom application and Event Viewer continued to get indexed. 

I started to notice an issue on one dashboard where no perfmon events were coming in. I reviewed another dashboard that checks the status of my forwarders and that's where I saw the updated installs. I went over the index the perfmon counters go to and validated only the hosts that were using Universal Forwarder 9.2.0.1 were coming in. 

My version of Enterprise was 9.2.1.0 and support recommended I update Enterprise to a newer version. After some testing, I went to Enterprise 9.3.5.0. Not ready for 9.4.X with trying to update the kvstore. Reviewing the Universal Forwarder compatibility matrix, I've kept my Universal Forwarders on 9.2.0.1, 9.2.6.0, and two were updated to 9.3.5.0. Updating Enterprise didn't correct the issue. 

I went through troubleshooting on the host looking over the config files. I did a rebuild of the resource counters and restarted the splunk forwarder service on one of the hosts using forwarder 9.2.6.0. 

I've looking at one of the hosts by adding the service account used as a local member of the administrators and Remote Management Users groups, adding a path variable for SPLUNK_HOME at "c:\program files\splunkuniversalforwarder". 

Chatted with the tech who pushed universal forwarder and they're not going to do that again. The hosts that got updated are members of my custom applications lower environments. I can live without the perfmon counters in the lower environments and none of my hosts in our production environment were updated. I know if I uninstall Forwarder and reinstall 9.2.0.1, the perfmon counters will resume coming in. 

Convinced its a change I need to do and thought I would check with the community who have updated their forwarders. I attached a copy of the inputs.conf from one of my hosts which is the same for all of them (aside from the environment name)







 

[PickleRick]

I see the useEnglishOnly setting which is known to cause problems. See my thread here https://community.splunk.com/t5/Getting-Data-In/Debugging-perfmon-input/m-p/621539#M107042

[PrewinThomas]

@DarthHerm 

Your inputs.conf looks good.

Check splunkd.log on the affected forwarder for errors related to perfmon or permissions
try upgrading - test the same config on one host with Universal Forwarder 9.3.5

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[livehybrid · Accepted solution]

Hi @DarthHerm 

Call me cynical but I suspect its a result of what has been done, rather than the Splunk upgrade files themselves, even rolling back the files might not correct things.

I think the first thing to double check is the file permissions, does the service account running Splunk have access to all the relevant files on the UF? How are your apps deployed to the UF? Is this via a DS or manual? Can you confirm the app is installed.

Are there any specific logs in the _internal index for one of these hosts, particularly anything that mentioned PerfMon! 

Based on the docs at https://help.splunk.com/en/splunk-enterprise/get-started/get-data-in/9.4/get-windows-data/monitor-wi... it seems important that the service user has "Performance Monitor Users" role - are you able to confirm this, please?

Another thing to double check - Can you run a btool ($SPLUNK_HOME\bin\splunk cmd btool inputs list --debug which should be a more detailed version of the inputs conf you provided. Has it loaded the relevant config in from your custom configuration?

Lastly, is environment_performance_logs and event index (rather than metric index)?

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[DarthHerm]

The issue has been identified. When the other agency pushed SplunkForwarder 9.2.6.0 to my hosts, NT SERVICE/SplunkForwarder was removed as a member of the performance monitor users’ group. That agency used Ivanti Patch for Endpoint Manager to push the updates.

With one of the hosts on 9.2.6.0, I kicked off the repair to SplunkForwarder and perfmon counters started to come in at the interval that was set for that host. I next moved to a powershell command to add NT SERVICE/SplunkForwarder back as a member of the performance monitor users group.

I asked the tech for a copy of the syntax used to push SplunkForwarder to my hosts to go over and validate.

I’m asking support about it too, has there been any known issues with Ivanti pushing SplunkForwarder updates?