Perfmon on Exchange App is not working: What does "unable to add counter" mean?

[btiggemann]

We have an environment with 4 Exchange Servers 2010 (2x CAS/HUB and 2x MBX), one each server we have installed splunk with the TA's for their appropriate roles. All four splunk instances on the exchange servers are forwarding there data to a single splunk instance. On the CAS servers we have the problem that perfmon is not generating any data because of several errors that are logged in splunkd.log. I have checked the perfmon.conf and its variables it looks ok. First on every time the perfmon script is running the following error occurs:

09-17-2012 15:59:09.179 +0200 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-perfmon.exe" -index perfmon" splunk-perfmon - Unable to add counter '\MSExchange ActiveSync(*)\Requests/sec' error 0xc0000bb9

09-17-2012 15:59:09.179 +0200 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-perfmon.exe" -index perfmon" splunk-perfmon - Perfmon - Invalid counter -

09-17-2012 15:59:09.179 +0200 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-perfmon.exe" -index perfmon" splunk-perfmon - Unable to add counter '\MSExchange ActiveSync(*)\Average Request Time' error 0xc0000bb9

09-17-2012 15:59:09.179 +0200 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-perfmon.exe" -index perfmon" splunk-perfmon - Perfmon - Invalid counter - e Non-Sm "\x84

09-17-2012 15:59:09.179 +0200 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-perfmon.exe" -index perfmon" splunk-perfmon - Unable to add counter '\ASP.NET(*)\Request Wait Time' error 0xc0000bb9

09-17-2012 15:59:09.179 +0200 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-perfmon.exe" -index perfmon" splunk-perfmon - Perfmon - Invalid counter -

09-17-2012 15:59:09.179 +0200 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-perfmon.exe" -index perfmon" splunk-perfmon - Unable to add counter '\ASP.NET(*)\Requests Current' error 0xc0000bb9

Can it be an issue with the installed .NET framework version? Currently version 4.0.30319 ist running. Exchange version is: 14.2.247.5

[btiggemann]

Ok, I have already done that. 🙂

[btiggemann]

Ok, I have already done that. 🙂

[btiggemann]

Hello Adrian,
I have used the perfon.exe from Microsoft to add the values that has been reported in the splunkd.log. Every counter was available and all of them are giving non-zero values, except those that are only giving a value grater than zero if there is a problem. So, it must be a problem with the splunk-perfmon.exe

Do you have any idea how we can go on with this?

Best regards
Benjamin

[ahall_splunk]

COntact your friendly Splunk support folks if you have a support contract. It needs more help than I can provide.

[ahall_splunk]

These errors indicate that the server cannot find the right Perfmon counters. They will be removed from the list of perfmon counters being checked. These errors will not prevent splunk-perfmon.exe from running.

Start up the regular perfmon.exe and use Add Counter to add the counters in and see if they are actually available and generating non-zero results. If you see something there, then the problem is with the splunk-perfmon collection and we need to dig further. If you do not see the counters and they are non-zero, then you need to investigate the Windows side of things.