Too many Events generated for Windows Security EventCode 4662 causing high resource issues like CPU

[sat94541]

The splunk service user seems to be reading the Active Directory's "Deleted Object Container" as frequent as 60 times per second.

Every time this container is accessed, it generates Windows Security audit logs; thus increasing the volume and frequency of Windows Security logs that the Universal Forwarder has to forward for indexing. We suspect this is the cause of the issue to spiral. Also, because of the increased log volume, the Security logs now rolls every 5 hours, instead of a few also we see high resource utaliation due to

[rbal_splunk · Accepted solution]

1. At some point, unknown root cause, the service account that the SplunkForwarder runs under began accessing the Directory Service object: CN=Deleted Objects,DC=DomainDnsZones,DC=global,DC=scj,DC=loc
   1. This object does not exist.
   2. DS Access auditing was enabled on the domain controllers.
   3. This caused the object access to record a 4662 event in the event log.
   4. The service account was reading this object dozens of times per second according to the event log.
   5. The security event log on the DC was configured to roll over at 4GB size which at its shortest only held 5.5 hours of data in the 4GB of size.
   6. Hence, the events in Splunk were no newer than 5.5 hours because the security log was filling faster than the forwarder could send and index the events.
   7. Since Splunk instance is set to dump event 4662 to a null queue so it was not indexed.
   8. DS Access object auditing was disabled and event 4662 stopped being logged.
   9. Within 12 hours the forwarder caught up to the event log.
   10. DS Access object auditing was enabled again. NO 4662 events are occurring. Somewhere along the way the forwarder decided to stop attempting to access that object.