Windows Event Log collection stops on Universal Forwarder while splunk-winevtlog.exe keeps running

[frknklnc]

Hello everyone,

We are investigating an intermittent Windows Event Log ingestion issue on multiple Windows Domain Controllers.

Affected channels:

- WinEventLog:Security
- WinEventLog:System
- WinEventLog:Application

Observed behavior:

- Windows Event Viewer continues to generate new events.
- The SplunkForwarder service remains running.
- The splunk-winevtlog.exe process remains running.
- Other inputs such as DNS Server and Directory Service may continue ingesting.
- Security, System, and Application ingestion stops after some time.
- Restarting the SplunkForwarder service temporarily restores ingestion, but the issue occurs again later.

Environment:

- Splunk Enterprise version: 9.4.12
- Universal Forwarder versions tested: 9.4.12, 10.2.4, and 10.4.0
- Windows Domain Controller servers
- Indexer Cluster environment

Important internal log message observed:

WinEventLogChannel::queryEvtChannel: Unable to set seek position to the given bookmark

Troubleshooting already performed:

- Verified that Windows Event Viewer continues generating events.
- Verified that the SplunkForwarder service and splunk-winevtlog.exe remain running.
- Tested Universal Forwarder versions 9.4.12, 10.2.4, and 10.4.0.
- Tested current_only = 1.
- Tested evt_resolve_ad_obj = 0.
- Cleared WinEventLog checkpoint and persistent storage.
- Increased Universal Forwarder maxKBps.
- Verified that indexer TCP input queues are not blocked.
- Tested an alternative Splunk receiving port instead of 9997.
- Confirmed that the SplunkForwarder service runs as Local System.
- Confirmed that EDR/antivirus is not blocking splunkd.exe or splunk-winevtlog.exe.

After upgrading to Universal Forwarder 10.4.0, the Security channel initially processed backlog events in burst mode, but later ingestion stopped again.

We are also working with Splunk Support and have provided DEBUG diagnostic logs.

Questions:

1. Has anyone experienced Windows Event Log channels stopping while splunk-winevtlog.exe remains running?
2. Did changing start_from, current_only, or checkpoint behavior resolve it in your environment?
3. Are there any known Windows Event Log, bookmark, or subscription-related workarounds for high-volume Domain Controllers?
4. Are there any specific Universal Forwarder versions or configuration recommendations that helped stabilize Security Event Log collection?

Any suggestions or similar experiences would be appreciated.

Thanks.

[PickleRick]

This might or might not be connected to your problem but I recall that windows eventlog inputs used to hit a performance limit at some point. A single input would get up to (in our case) around 12kEPS and wouldn't go pass that threshold. So it would start lagging behind during the day and would catch up during off-hours. No tweaking on UF's side (most importantly, adding pipelines) helped since the issue seemed to be with a single input, not UF as a whole - other inputs were processed OK. And - most importantly - it wasn't related to overall server performance since we had plenty of CPU left (and if I remember correctly, we weren't even hogging a single cpu up to 100%).

Luckily for us, we weren't ingesting locally generated events but those from Forwarded Events from WEF subscriptions. So after a bit of bending over backwards we created more event log channels and split the subscriptions across those channels so a single input wouldn't get overwhelmed.

If your underlying problem is the same, the input might be lagging up to a point where the eventlog file gets rotated on OS side and the event pointer is no longer valid. Raising eventlog file size could alleviate the problem a little by giving the input a bigger buffer to work with but of course wouldn't remediate it completely.

[frknklnc]

Thanks for sharing this. This sounds relevant to our case.

We are seeing similar behavior where the Universal Forwarder and splunk-winevtlog.exe remain running, other inputs may continue ingesting, but high-volume Windows Event Log channels such as Security stop progressing or fall behind.

We also observed significant delay on the Security channel. In one test window, the average delay was around 28 minutes, p95 was around 56 minutes, and max delay was close to 60 minutes. We are also seeing the following internal error:

WinEventLogChannel::queryEvtChannel: Unable to set seek position to the given bookmark

Your point about the input lagging behind until the Windows Event Log rolls/overwrites and the bookmark becoming invalid is interesting and may explain the bookmark behavior.

We will check the Security EPS rate, delay trend, and Windows Event Log retention/rollover behavior to see if the input is falling behind during high-volume periods.

In our case these are locally generated Domain Controller Security logs, not WEF Forwarded Events, so splitting WEF subscriptions is not directly applicable. But the single-channel throughput limitation theory is very useful.

Thanks again for the insight.