Not receiving performance for a DC

[lfcowart]

I believe I configured them exactly the same (took the Splunk_TA_windows and TA-DomainController-NT6 defaults and copied them into %splunk_home%\etc\apps on 2 of my DCs. Both are 2008 R2. I applied a GPO to the default dc policy to enable the auditing and powershell as outlined in the doc.

However, one of my DCs does not report any data in the Directory Services Performance and Replication Performance sections. I ran a search manually for search eventtype=perfmon-ntds host="hostname" and it returns nothing for the problem DC.

I'm not really sure where to look to troubleshoot this now.

[diegosainz]

I have set the variable as mentioned above and I still am not getting perfmon data. But I do get it from the TA-Windows add on.

[jchampagne]

I was having this same problem on my Windows hosts, I tried to manually run splunk-perfmon.exe to see what the issue was. It turns out SPLUNK_HOME was not set. I set the system variable and it started working.

[lfcowart · Accepted solution]

So I'm pretty sure I figured it out. I was missing all of my NTDS performance counters (similar to http://blogs.technet.com/b/brad_rutkowski/archive/2009/03/19/ntds-performance-counters-missing.aspx).

Pretty strange...

Exporting the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Performance from the working DC and importing it to the broken, then running LODCTR /R and then restarting splunk got the performance data to populate.

[ahall_splunk]

Goods to know this is common enough to write a blog article about. I was going to get you to go into Perfmon next and see if the NTDS objects were actually there.

[ahall_splunk]

I believe there is a bug in the v1.0 version of the Splunk App for Active Directory that is fixed in the v1.1 version that is coming out next week. The bug is "perfmon.conf file does not place performance events in the right index".

To verify this, check the main index for your perfmon data. You can do this by searching in the Search App for index=main source="Perfmon:NTDS"

If this is the case, then edit the default/perfmon.conf file within the TA-DomainController-NT6 on your domain controller. Add a line that says "index=perfmon" to each stanza. This will cause the perfmon data to go to the right place. As I mentioned earlier, this is the exact change that v1.1 brings to this app.

[lfcowart]

This is the contents of my \etc\apps\TA-DomainController-NT6\default\perfmon.conf

[PERFMON:Processor]
object = Processor
counters = *
instances = *
interval = 10
disabled = 0

[PERFMON:Memory]
object = Memory
counters = *
interval = 10
disabled = 0

[PERFMON:Network_Interface]
object = Network Interface
counters = *
instances = *
interval = 10
disabled = 0

[PERFMON:DFS_Replicated_Folders]
object = DFS Replicated Folders
counters = *
instances = *
interval = 30
disabled = 0

[PERFMON:NTDS]
object = NTDS
counters = *
interval = 10
disabled = 0

There's no ..\local\perfmon.conf

[lfcowart]

At first it said SPLUNK_HOME must be set, so i set it to C:\Program Files\SplunkUniversalForwarder\

Then I get this output from running splunk-perfmon.exe:

ERROR splunk-perfmon - The object specified - 'NTDS' in stanza - 'NTDS' in conf
file is not valid.

then it gives me network_interface and processor performances.

[lfcowart]

sorry didn't see your reply. let me try that.

[lfcowart]

I checked the splunkd.log after restarting, and I got this line. not sure if it's a problem or which of the conf files it's referring to...

08-09-2012 11:18:26.963 -0400 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -index perfmon" splunk-perfmon - The object specified - 'NTDS' in stanza - 'NTDS' in conf file is not valid.

[ahall_splunk]

This is starting to sound like a problem with the splunk-perfmon scripted input. Can you run splunk-perfmon.exe manually and see if it even runs on the broken host?

[lfcowart]

Also:

index="main" host="brokenhost" doesn't give me any Perfmon:NTDS results

[lfcowart]

So I ran the search and it didn't return anything.

I ran:

index="perfmon" source="Perfmon:NTDS" and have results only from 1 host and not from the one that seems to be broken.

If I do

index="perfmon" host="brokenhost"
I get Perfmon:Network_Interface and Perfmon:Processor but nothing for NTDS