Why is the Splunk Universal Forwarder on my domain controllers consuming 100% CPU with error "DsBind failed"?

[trademarq]

On more than a few of my domain controllers, the Splunk Universal Forwarder is consuming 100% CPU and spewing many errors in splunkd.log like this:

06-22-2015 15:26:58.603 -0400 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe (/splunk-winevtlog.exe)"" splunk-winevtlog - EvtDC::connectToDC: DsBind failed: (5)

This appears to be an issue with the SID resolution as I am collecting Windows Logs on these domain controllers. I'm aware of the evt_dc_name parameter in inputs.conf, but I don't wish to use it because the objects should all be available locally. How do I resolve this issue?

[trademarq · Accepted solution]

I was able to confirm that a security control (Symantec Critical Server Protection / DSP) was preventing the Splunk service from doing what it wanted to do. Resolving the security rules fixed the issue.

[acharlieh]

According to MSDN RPC error code 5 is ERROR_ACCESS_DENIED which definitely gives credence to @dolivasoh's theory of this being a problem that could easily land one in the 7th circle. Are you running the UF as a domain user account? There's also discussion about what user you should run Splunk as on Windows and what permissions said user should have at a base level in the docs.

[dolivasoh]

UniversalForwarder+Windows-Permissions=HELL

Make sure you have adequate permissions to do all things specified on the forwarder. Not a complete solution but a good place to start.

[trademarq]

Running Splunk 6.2.0 Forwarder in most cases, will upgrade to a newer revision if that is a confirmed fix.