App for Exchange IIS Log Times

[bstricklin]

I have installed the app for Exchange on our Exchange 2010 system. There are two mailbox servers and one CAS/HT server. All the logs appear to be in Splunk and all mailbox related information appeared the the app's page. However none of the CAS or HT related information is being displayed. If I do a simple search for the CAS/HT host all the logs are there.

One problem I noticed is the IIS logs are not displaying the correct time. They are 5 hours in the future. I checked the fwd_win2008r2_iis/default/props.conf file on the CAS/HT server and the TZ = GMT line is there. The server is in the GMT-5 timezone, and all non-IIS logs appear to have the correct time. Also, all logs are in the default locations on the server and the main index is being used, so I have not changed the fwd_wind2008r2_iis, fwd_exchange2010_cas, or fwd_exchange2010_hub .conf files from the default.

UPDATE:
I have found some errors in the foward's logs. It looks like some of the performance monitors in the app's conf files do not exist on this server. Is this app compatible with Exchange 2010 SP1?

UPDATE:
I have found the same performance monitor errors in the logs for the forwarder on the mailbox servers also. Below is a copy of the relevant lines from the mailbox server.

11-18-2011 16:14:54.368 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui" splunk-perfmon - Unable to add counter '\MSExchange Assistants()\Average Event Processing Time in Seconds' error 0xc0000bb8
11-18-2011 16:14:54.368 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui"
ERROR splunk-perfmon - Perfmon - Invalid counter -
11-18-2011 16:14:54.368 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui" splunk-perfmon - Unable to add counter '\MSExchange Assistants()\Average Event Queue Time in seconds' error 0xc0000bb8
11-18-2011 16:14:54.368 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui"
ERROR splunk-perfmon - Perfmon - Invalid counter -
11-18-2011 16:14:54.368 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui" splunk-perfmon - Unable to add counter '\MSExchange Assistants()\Average Mailbox Processing Time In seconds' error 0xc0000bb8
11-18-2011 16:14:54.368 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui"
ERROR splunk-perfmon - Perfmon - Invalid counter -
11-18-2011 16:14:54.368 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui" splunk-perfmon - Unable to add counter '\MSExchange Assistants()\Events Polled/sec' error 0xc0000bb8
11-18-2011 16:14:54.368 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui"
ERROR splunk-perfmon - Perfmon - Invalid counter -
11-18-2011 16:14:54.368 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui" splunk-perfmon - Unable to add counter '\MSExchange Assistants()\Mailboxes processed/sec' error 0xc0000bb8
11-18-2011 16:14:54.368 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui"
ERROR splunk-perfmon - Perfmon - Invalid counter -
11-18-2011 16:14:54.368 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui" splunk-perfmon - Unable to add counter '\MSExchange Assistants()\Number of Failed Event Dispatchers' error 0xc0000bb8
11-18-2011 16:14:54.368 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui"
ERROR splunk-perfmon - Perfmon - Invalid counter -
11-18-2011 16:14:54.368 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui" splunk-perfmon - Unable to add counter '\MSExchange Assistants()\Percentage of Failed Event Dispatchers' error 0xc0000bb8
11-18-2011 16:14:54.368 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui"
ERROR splunk-perfmon - Perfmon - Invalid counter -
11-18-2011 16:14:54.993 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui" splunk-perfmon - Unable to add counter '\MSExchange Replication()\ReplayGenerationsPerMinute' error 0xc0000bb9
11-18-2011 16:14:54.993 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui"
ERROR splunk-perfmon - Perfmon - Invalid counter -
11-18-2011 16:14:54.993 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui" splunk-perfmon - Unable to add counter '\MSExchange Replication()\ReplayGenerationsPerMinute' error 0xc0000bb9
11-18-2011 16:14:54.993 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui"
ERROR splunk-perfmon - Perfmon - Invalid counter -
11-18-2011 16:14:54.993 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui" splunk-perfmon - Unable to add counter '\MSExchange Replication()\ReplayGenerationsPerMinute' error 0xc0000bb9
11-18-2011 16:14:54.993 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui"
ERROR splunk-perfmon - Perfmon - Invalid counter -
11-18-2011 16:14:54.993 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui" splunk-perfmon - Unable to add counter '\MSExchange Replication()\ReplayGenerationsPerMinute' error 0xc0000bb9
11-18-2011 16:14:54.993 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui"
ERROR splunk-perfmon - Perfmon - Invalid counter -
11-18-2011 16:14:54.993 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui" splunk-perfmon - Unable to add counter '\MSExchange Replication()\ReplayGenerationsPerMinute' error 0xc0000bb9
11-18-2011 16:14:54.993 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui"
ERROR splunk-perfmon - Perfmon - Invalid counter -
11-18-2011 16:14:54.993 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui" splunk-perfmon - Unable to add counter '\MSExchange Replication()\ReplayGenerationsPerMinute' error 0xc0000bb9
11-18-2011 16:14:54.993 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui"
ERROR splunk-perfmon - Perfmon - Invalid counter -
11-18-2011 16:14:54.993 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui" splunk-perfmon - Unable to add counter '\MSExchange Replication()\ReplayGenerationsPerMinute' error 0xc0000bb9
11-18-2011 16:14:54.993 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui"
ERROR splunk-perfmon - Perfmon - Invalid counter -
11-18-2011 16:14:54.993 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui" splunk-perfmon - Unable to add counter '\MSExchange Replication(*)\ReplayGenerationsPerMinute' error 0xc0000bb9
11-18-2011 16:14:54.993 -0500 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -noui"
ERROR splunk-perfmon - Perfmon - Invalid counter -

[ahall_splunk]

There are two issues here:

1) The logs are not being recognized. You can use the search bar in the Exchange app to search for eventtype=client-iis-logs - it should produce results. If that works, try searching for eventtype=client-owa-usage to see just the OWA information. If these produce data, the dashboards should be populated as well. If the former (client-iis-logs) does not produce data, then it's likely to be an IIS log extraction problem - ensure the MSWindows:*:IIS sourcetype is in the msexchange index or the eventtype definition (in eventtypes.conf) points to the right place. Also, ensure the fields (cs_username, cs_uri_path, etc.) are being extracted. If it isn't, then the transforms.conf in both the TA-Windows-2008R2-IIS and the Splunk_for_Exchange app needs to be altered to match the actual log format.

2) Perfmon is not being pulled properly. Microsoft likes to rename perfmon counters on a regular basis. The actual perfmon counter names are stored in perfmon.conf and used through out the app for plotting. However, they are also documented on technet, which is what we used for this gathering. Use perfmon.exe to get the real names from your system. If your system is not producing specific perfmon counters, remove them from perfmon.conf. (The logs saying Invalid counter are annoying, but harmless, and not counted towards your license usage).

As always, if you have a support contract, feel free to open up a support case. Splunk App for Microsoft Exchange is a supported app.