Upgraded Windows Domain Controllers from 2008 R2 to 2012 R2, why are 6.2.2 and 6.2.3 universal forwarders not forwarding Security Event Logs?

[AndreaEClark]

My Help Desk relies upon using the Splunk server to assist with identifying the source machine or BYOD for account lockouts. Since we've rebuilt our servers on heartier platforms, (same names, same IP addresses, added resources to our DC VMs) none of my 2012 R2 machines are logging Windows Security Log events.

Since the DCs have been upgraded to 2012 R2, none of them are logging security event logs. The servers logged security events into indexes I created when they were still at 2008 R2. As far as the VMs, we extended system partition and allocated more procs and ram. Hardware devices where demoted and new device promoted.

My Domain Controller GPOs for Advanced Audit Configuration, hasn't changed since we upgraded the DCs from 2008 R2 to 2012 R2.

Advanced Audit Configuration
Account Logon
Policy Setting
Audit Credential Validation Success, Failure
Audit Kerberos Authentication Service Success, Failure
Audit Kerberos Service Ticket Operations Success, Failure
Audit Other Account Logon Events Success, Failure
Account Management
Policy Setting
Audit Application Group Management Success, Failure
Audit Computer Account Management Success, Failure
Audit Distribution Group Management Success, Failure
Audit Other Account Management Events Success, Failure
Audit Security Group Management Success, Failure
Audit User Account Management Success, Failure
Detailed Tracking
Policy Setting
Audit Process Creation Success, Failure
Audit Process Termination Success, Failure
Audit RPC Events Success, Failure
DS Access
Policy Setting
Audit Directory Service Access Success, Failure
Audit Directory Service Changes Success, Failure
Logon/Logoff
Policy Setting
Audit Account Lockout Success, Failure
Audit Logoff Success, Failure
Audit Logon Success, Failure
Audit Other Logon/Logoff Events Success, Failure
Audit Special Logon Success, Failure
Object Access
Policy Setting
Audit Application Generated Success, Failure
Audit Certification Services Success, Failure
Audit File Share Success, Failure
Audit File System Success, Failure
Audit Kernel Object Success, Failure
Audit Registry Success, Failure
Policy Change
Policy Setting
Audit Audit Policy Change Success, Failure
Audit Authentication Policy Change Success, Failure
Audit Authorization Policy Change Success, Failure
Audit Other Policy Change Events Failure
Privilege Use
Policy Setting
Audit Sensitive Privilege Use Success, Failure
System
Policy Setting
Audit IPsec Driver Success, Failure
Audit Other System Events Failure
Audit Security State Change Success, Failure
Audit Security System Extension Success, Failure
Audit System Integrity Success, Failure

[AndreaEClark · Accepted solution]

6.2.2 and 6.2.3 forwarder bundles the Windows TA. In %SplunkUniversalForwarder%/etc/apps/Splunk_TA_windows/default/inputs.conf there are the following stanzas

[WinEventLog://Application]
disabled = 1
start_from = oldest
current_only = 0
checkpointInterval = 5
index = wineventlog
renderXml=false

[WinEventLog://Security]
disabled = 1
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
blacklist1 = EventCode="4662" Message="Object Type:\s+(?!groupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:\s+(?!groupPolicyContainer)"
index = wineventlog
renderXml=false

[WinEventLog://System]
disabled = 1
start_from = oldest
current_only = 0
checkpointInterval = 5
index = wineventlog
renderXml=false

When you enable predefined eventlogs such as Security and System in the forwarder .msi installer, the stanzas in %SplunkUniversalForwarder/etc/apps/Splunk_TA_windows/local/inputs.conf looks like this

[WinEventLog://Security]
disabled = 0

[WinEventLog://System]
disabled = 0

The Windows TA had not been installed on the Splunk Indexer/Searcher, therefore Splunk didn't know where to put the data. The Windows TA was then installed on the Splunk Indexer/Searcher and data began showing up in the wineventlog index.

I have since edited these files to point to a user defined index. We are now happily Splunking Windows Security data again. 🙂

[leochan]

Are you pulling the event log? By eventID? Can you post your query?

[AndreaEClark]

Using the following syntax. With or without quotes, the last indexed data was from 6/1 just prior to removing the last 2008 R2 domain controller from our environment.

index=* sourcetype="WinEventLog:Security"

[AndreaEClark]

This looks like a forwarder bug on Windows OS 2012 R2. I am receiving Active Directory data from the forwarder. When I installed/reinstalled the client I selected Security log, System log and Active Directory data.

Anyone else attempted to deploy at 6.2.x Universal Forwarder to Windows 2012 R2 Domain Controllers that are forwarding to a 6.2.2. Splunk Server and not getting security log data?

[AndreaEClark]

Further analysis has revealed that the security logs are actually being sent to the Splunk indexer, but it is not indexing the data as a source=WinEventlog:Security or sourcetype=WinEventlog:Security. The only data in indexer is indexing is source=ActiveDirectory or sourcetype=ActiveDirectory.

I'll post more as we discover more.