Windows Universal Forwarders no longer performing AD GUID/SID-to-value lookups after upgrade to 8.1

[abaumbusch]

Greetings!

We recently upgraded our UFs throughout the environment to 8.1.0, and since the upgrade, none of the Windows based forwarders appear to be doing AD GUID/SID-to-value lookups. We have verified that  evt_resolve_ad_obj = 1 is set in inputs.conf for the [WinEventLog://Security] stanza (verified with btool as well), and prior to the upgrade, the functionality was working fine. We tried installing the 8.1.1 version of the forwarder on one box as a test, but the problem persisted. Has anyone seen this or have any suggestions on what to check?

This is a multi-site clustered environment running Splunk Enterprise 8.0.7. Thanks for your help!

[mmatturro]

Looks like this is still an issue in 8.1.2: https://docs.splunk.com/Documentation/Forwarder/8.1.2/Forwarder/KnownIssues.

[jeff · Accepted solution]

I was told the fix may come in 8.1.3. I can confirm adding  use_old_eventlog_api = 1 to the wineventlog:security stanza in inputs conf seems to resolve the issue. I deployed this change to forwarders on test and dev systems in our environment and the sids were properly resolved.

[pbarbuto]

Do you know if using use_old_eventlog_api caused any other issues or side effects? Any issues with field extractions?

[jeff]

The following has been added to known issues:

SPL-199409, SPL-199691: Windows EventLog SIDs no longer resolving after upgrade to 8.1 

Splunk Devs acknowledged the bug (I work with @abaumbusch) and provided the following workarounds:

• replace the splunk-winevtlog.exe binary on an 8.1.x forwarder with a copy from an 8.0.x forwarder, OR
• set 'use_old_eventlog_api' to true for any WinEventLogs that need the SID/GUID translation. It was found that this issue does not affect any channel where 'use_old_eventlog_api' is enabled.

[abaumbusch]

I opened a support case with Splunk, but research on their end is still ongoing. They thought the issue might be with Windows machines on older versions of the OS (e.g. 2012), but we are seeing this on 2016 servers as well. If I do get a solutions from them, I will make sure to post it. Thanks for your response!

[greggernigant]

Thanks!

If that helps, in my environment the issue exists on server 2016 and 2019, tried ufw 8.1.0 and the latest 8.1.1 (splunkforwarder-8.1.1-08187535c166-x64-release) without any luck.

Cheers.

[greggernigant]

Have you managed to resolve this?

I have this exact same problem, unfortunately haven't found any other solution than downgrading to 8.0.x

Enabled DEBUG logging but can't find any indication of what is failing, following this article here https://docs.splunk.com/Documentation/Splunk/8.1.1/Data/MonitorWindowseventlogdata suggest 

"If you discover that SIDs are not being translated properly, you can review %SPLUNK_HOME%\var\log\splunkd.log for clues on what the problem might be. Problems with SID translations appear in the DsCrackNamesW API, which appear at the DEBUG logging level for splunkd.log, in the ExecProcessor log facility. For information on how to set the DEBUG logging level to see debug logs, see Enable debug logging in the Troubleshooting Manual." 

Any suggestions appreciated!