Getting Data In

Forwarder shows extreme lag or latency when sending Windows Security Eventlog data

hexx
Splunk Employee
Splunk Employee

A Windows 2008R2 Domain Controller in another geographical, and the Security Events are perpetually multiple days behind.

There's no limit on outgoing forwarder throughput; and watching a local file on the DC with the forwarder works fine, with no notable latency sending data to the indexers.

Clearing the Windows Security log allowed the events to catch-up for a short while, but they quickly fell behind again.
The other Event Log channels were turned off, and that didin't help either.

1 Solution

ekost
Splunk Employee
Splunk Employee

The Windows Security Event Log monitor has the setting "evt_resolve_ad_obj" enabled by default. This allows for the resolution of Active Directory objects like Globally Unique IDentifier (GUID) and Security IDentifier (SID) objects to their
canonical names. Unfortunately, sometimes the closest DC available to resolve those objects is not found, and the forwarder will grab at another DC. This can result in excessive lag in Security events while the Splunk Forwarder is waiting
on the DC to provide the SID/GUID. The log entries are seen in splunkd.log during the forwarder startup:

INFO  WinEventLogChannel - init: Binding to DC to translate guids/sids for channel='Security'
INFO  WinEventLogChannel - EvtDC::bind: Found DC='\\elric-dc01.oroboros.net', DCsite='CentralCity-HQ', ClientSite = 'Resembool', Domain='oroboros.net'
WARN  WinEventLogChannel - Failed to retrieve the PDC in the closest site.
INFO  WinEventLogChannel - init: Successfully bound to DC, dc_bind_time=2469 msec
INFO  WinEventLogInputProcessor - main-thread: Processing existing Windows Event Log 'Security'

For the Security channel it's not recommended that the setting "evt_resolve_ad_obj" be disabled. Please use the complementary options evt_dc_name and evt_dns_name in inputs.conf to
explicitly define the DC and if necessary DNS host to use.

[WinEventLog:Security]
## Forwarder is installed on my DC, so localhost is appropriate
evt_dc_name = localhost
## Default
evt_resolve_ad_obj = 1

Restart Splunk services for the changes to take effect.

View solution in original post

rbal_splunk
Splunk Employee
Splunk Employee

In one of the setup we found that after up running more than 9 months, Windows system performance is degraded and event log system is not responsive. Restarting Windows system is the workaround.

0 Karma

ekost
Splunk Employee
Splunk Employee

The Windows Security Event Log monitor has the setting "evt_resolve_ad_obj" enabled by default. This allows for the resolution of Active Directory objects like Globally Unique IDentifier (GUID) and Security IDentifier (SID) objects to their
canonical names. Unfortunately, sometimes the closest DC available to resolve those objects is not found, and the forwarder will grab at another DC. This can result in excessive lag in Security events while the Splunk Forwarder is waiting
on the DC to provide the SID/GUID. The log entries are seen in splunkd.log during the forwarder startup:

INFO  WinEventLogChannel - init: Binding to DC to translate guids/sids for channel='Security'
INFO  WinEventLogChannel - EvtDC::bind: Found DC='\\elric-dc01.oroboros.net', DCsite='CentralCity-HQ', ClientSite = 'Resembool', Domain='oroboros.net'
WARN  WinEventLogChannel - Failed to retrieve the PDC in the closest site.
INFO  WinEventLogChannel - init: Successfully bound to DC, dc_bind_time=2469 msec
INFO  WinEventLogInputProcessor - main-thread: Processing existing Windows Event Log 'Security'

For the Security channel it's not recommended that the setting "evt_resolve_ad_obj" be disabled. Please use the complementary options evt_dc_name and evt_dns_name in inputs.conf to
explicitly define the DC and if necessary DNS host to use.

[WinEventLog:Security]
## Forwarder is installed on my DC, so localhost is appropriate
evt_dc_name = localhost
## Default
evt_resolve_ad_obj = 1

Restart Splunk services for the changes to take effect.

aelliott
Motivator

We did as you said above and everything started coming in great.. a little too great.. nearly a million events per hour.. So we found a lot of 4662 event code events and ultimately stumbled upon the following post: http://blogs.splunk.com/2014/05/23/controlling-4662-messages-in-the-windows-security-event-log/

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...