All Apps and Add-ons

Not receiving performance for a DC

Path Finder

I believe I configured them exactly the same (took the Splunk_TA_windows and TA-DomainController-NT6 defaults and copied them into %splunk_home%\etc\apps on 2 of my DCs. Both are 2008 R2. I applied a GPO to the default dc policy to enable the auditing and powershell as outlined in the doc.

However, one of my DCs does not report any data in the Directory Services Performance and Replication Performance sections. I ran a search manually for search eventtype=perfmon-ntds host="hostname" and it returns nothing for the problem DC.

I'm not really sure where to look to troubleshoot this now.

0 Karma
1 Solution

Path Finder

So I'm pretty sure I figured it out. I was missing all of my NTDS performance counters (similar to http://blogs.technet.com/b/brad_rutkowski/archive/2009/03/19/ntds-performance-counters-missing.aspx).

Pretty strange...

Exporting the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Performance from the working DC and importing it to the broken, then running LODCTR /R and then restarting splunk got the performance data to populate.

View solution in original post

Path Finder

I have set the variable as mentioned above and I still am not getting perfmon data. But I do get it from the TA-Windows add on.

0 Karma

Path Finder

I was having this same problem on my Windows hosts, I tried to manually run splunk-perfmon.exe to see what the issue was. It turns out SPLUNK_HOME was not set. I set the system variable and it started working.

0 Karma

Path Finder

So I'm pretty sure I figured it out. I was missing all of my NTDS performance counters (similar to http://blogs.technet.com/b/brad_rutkowski/archive/2009/03/19/ntds-performance-counters-missing.aspx).

Pretty strange...

Exporting the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Performance from the working DC and importing it to the broken, then running LODCTR /R and then restarting splunk got the performance data to populate.

View solution in original post

Splunk Employee
Splunk Employee

Goods to know this is common enough to write a blog article about. I was going to get you to go into Perfmon next and see if the NTDS objects were actually there.

Splunk Employee
Splunk Employee

I believe there is a bug in the v1.0 version of the Splunk App for Active Directory that is fixed in the v1.1 version that is coming out next week. The bug is "perfmon.conf file does not place performance events in the right index".

To verify this, check the main index for your perfmon data. You can do this by searching in the Search App for index=main source="Perfmon:NTDS"

If this is the case, then edit the default/perfmon.conf file within the TA-DomainController-NT6 on your domain controller. Add a line that says "index=perfmon" to each stanza. This will cause the perfmon data to go to the right place. As I mentioned earlier, this is the exact change that v1.1 brings to this app.

0 Karma

Path Finder

This is the contents of my \etc\apps\TA-DomainController-NT6\default\perfmon.conf

[PERFMON:Processor]
object = Processor
counters = *
instances = *
interval = 10
disabled = 0

[PERFMON:Memory]
object = Memory
counters = *
interval = 10
disabled = 0

[PERFMON:Network_Interface]
object = Network Interface
counters = *
instances = *
interval = 10
disabled = 0

[PERFMON:DFS_Replicated_Folders]
object = DFS Replicated Folders
counters = *
instances = *
interval = 30
disabled = 0

[PERFMON:NTDS]
object = NTDS
counters = *
interval = 10
disabled = 0

There's no ..\local\perfmon.conf

0 Karma

Path Finder

At first it said SPLUNK_HOME must be set, so i set it to C:\Program Files\SplunkUniversalForwarder\

Then I get this output from running splunk-perfmon.exe:

ERROR splunk-perfmon - The object specified - 'NTDS' in stanza - 'NTDS' in conf
file is not valid.

then it gives me network_interface and processor performances.

0 Karma

Path Finder

sorry didn't see your reply. let me try that.

0 Karma

Path Finder

I checked the splunkd.log after restarting, and I got this line. not sure if it's a problem or which of the conf files it's referring to...

08-09-2012 11:18:26.963 -0400 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe" -index perfmon" splunk-perfmon - The object specified - 'NTDS' in stanza - 'NTDS' in conf file is not valid.

0 Karma

Splunk Employee
Splunk Employee

This is starting to sound like a problem with the splunk-perfmon scripted input. Can you run splunk-perfmon.exe manually and see if it even runs on the broken host?

0 Karma

Path Finder

Also:

index="main" host="brokenhost" doesn't give me any Perfmon:NTDS results

0 Karma

Path Finder

So I ran the search and it didn't return anything.

I ran:

index="perfmon" source="Perfmon:NTDS" and have results only from 1 host and not from the one that seems to be broken.

If I do

index="perfmon" host="brokenhost"
I get Perfmon:Network_Interface and Perfmon:Processor but nothing for NTDS

0 Karma