Hello everyone, We are investigating an intermittent Windows Event Log ingestion issue on multiple Windows Domain Controllers. Affected channels: - WinEventLog:Security - WinEventLog:System - WinEventLog:Application Observed behavior: - Windows Event Viewer continues to generate new events. - The SplunkForwarder service remains running. - The splunk-winevtlog.exe process remains running. - Other inputs such as DNS Server and Directory Service may continue ingesting. - Security, System, and Application ingestion stops after some time. - Restarting the SplunkForwarder service temporarily restores ingestion, but the issue occurs again later. Environment: - Splunk Enterprise version: 9.4.12 - Universal Forwarder versions tested: 9.4.12, 10.2.4, and 10.4.0 - Windows Domain Controller servers - Indexer Cluster environment Important internal log message observed: WinEventLogChannel::queryEvtChannel: Unable to set seek position to the given bookmark Troubleshooting already performed: - Verified that Windows Event Viewer continues generating events. - Verified that the SplunkForwarder service and splunk-winevtlog.exe remain running. - Tested Universal Forwarder versions 9.4.12, 10.2.4, and 10.4.0. - Tested current_only = 1. - Tested evt_resolve_ad_obj = 0. - Cleared WinEventLog checkpoint and persistent storage. - Increased Universal Forwarder maxKBps. - Verified that indexer TCP input queues are not blocked. - Tested an alternative Splunk receiving port instead of 9997. - Confirmed that the SplunkForwarder service runs as Local System. - Confirmed that EDR/antivirus is not blocking splunkd.exe or splunk-winevtlog.exe. After upgrading to Universal Forwarder 10.4.0, the Security channel initially processed backlog events in burst mode, but later ingestion stopped again. We are also working with Splunk Support and have provided DEBUG diagnostic logs. Questions: 1. Has anyone experienced Windows Event Log channels stopping while splunk-winevtlog.exe remains running? 2. Did changing start_from, current_only, or checkpoint behavior resolve it in your environment? 3. Are there any known Windows Event Log, bookmark, or subscription-related workarounds for high-volume Domain Controllers? 4. Are there any specific Universal Forwarder versions or configuration recommendations that helped stabilize Security Event Log collection? Any suggestions or similar experiences would be appreciated. Thanks.
... View more
