Forwarder shows extreme lag or latency when sending Windows Security Eventlog data

[hexx]

A Windows 2008R2 Domain Controller in another geographical, and the Security Events are perpetually multiple days behind.

There's no limit on outgoing forwarder throughput; and watching a local file on the DC with the forwarder works fine, with no notable latency sending data to the indexers.

Clearing the Windows Security log allowed the events to catch-up for a short while, but they quickly fell behind again.
The other Event Log channels were turned off, and that didin't help either.

[rbal_splunk]

In one of the setup we found that after up running more than 9 months, Windows system performance is degraded and event log system is not responsive. Restarting Windows system is the workaround.

[ekost · Accepted solution]

The Windows Security Event Log monitor has the setting "evt_resolve_ad_obj" enabled by default. This allows for the resolution of Active Directory objects like Globally Unique IDentifier (GUID) and Security IDentifier (SID) objects to their
canonical names. Unfortunately, sometimes the closest DC available to resolve those objects is not found, and the forwarder will grab at another DC. This can result in excessive lag in Security events while the Splunk Forwarder is waiting
on the DC to provide the SID/GUID. The log entries are seen in splunkd.log during the forwarder startup:

INFO  WinEventLogChannel - init: Binding to DC to translate guids/sids for channel='Security'
INFO  WinEventLogChannel - EvtDC::bind: Found DC='\\elric-dc01.oroboros.net', DCsite='CentralCity-HQ', ClientSite = 'Resembool', Domain='oroboros.net'
WARN  WinEventLogChannel - Failed to retrieve the PDC in the closest site.
INFO  WinEventLogChannel - init: Successfully bound to DC, dc_bind_time=2469 msec
INFO  WinEventLogInputProcessor - main-thread: Processing existing Windows Event Log 'Security'

For the Security channel it's not recommended that the setting "evt_resolve_ad_obj" be disabled. Please use the complementary options evt_dc_name and evt_dns_name in inputs.conf to
explicitly define the DC and if necessary DNS host to use.

[WinEventLog:Security]
## Forwarder is installed on my DC, so localhost is appropriate
evt_dc_name = localhost
## Default
evt_resolve_ad_obj = 1

Restart Splunk services for the changes to take effect.

[aelliott]

We did as you said above and everything started coming in great.. a little too great.. nearly a million events per hour.. So we found a lot of 4662 event code events and ultimately stumbled upon the following post: http://blogs.splunk.com/2014/05/23/controlling-4662-messages-in-the-windows-security-event-log/