Windows Event Log collection stops on Universal Forwarder while splunk-winevtlog.exe keeps running

[frknklnc]

Hello everyone,

We are investigating an intermittent Windows Event Log ingestion issue on multiple Windows Domain Controllers.

Affected channels:

- WinEventLog:Security
- WinEventLog:System
- WinEventLog:Application

Observed behavior:

- Windows Event Viewer continues to generate new events.
- The SplunkForwarder service remains running.
- The splunk-winevtlog.exe process remains running.
- Other inputs such as DNS Server and Directory Service may continue ingesting.
- Security, System, and Application ingestion stops after some time.
- Restarting the SplunkForwarder service temporarily restores ingestion, but the issue occurs again later.

Environment:

- Splunk Enterprise version: 9.4.12
- Universal Forwarder versions tested: 9.4.12, 10.2.4, and 10.4.0
- Windows Domain Controller servers
- Indexer Cluster environment

Important internal log message observed:

WinEventLogChannel::queryEvtChannel: Unable to set seek position to the given bookmark

Troubleshooting already performed:

- Verified that Windows Event Viewer continues generating events.
- Verified that the SplunkForwarder service and splunk-winevtlog.exe remain running.
- Tested Universal Forwarder versions 9.4.12, 10.2.4, and 10.4.0.
- Tested current_only = 1.
- Tested evt_resolve_ad_obj = 0.
- Cleared WinEventLog checkpoint and persistent storage.
- Increased Universal Forwarder maxKBps.
- Verified that indexer TCP input queues are not blocked.
- Tested an alternative Splunk receiving port instead of 9997.
- Confirmed that the SplunkForwarder service runs as Local System.
- Confirmed that EDR/antivirus is not blocking splunkd.exe or splunk-winevtlog.exe.

After upgrading to Universal Forwarder 10.4.0, the Security channel initially processed backlog events in burst mode, but later ingestion stopped again.

We are also working with Splunk Support and have provided DEBUG diagnostic logs.

Questions:

1. Has anyone experienced Windows Event Log channels stopping while splunk-winevtlog.exe remains running?
2. Did changing start_from, current_only, or checkpoint behavior resolve it in your environment?
3. Are there any known Windows Event Log, bookmark, or subscription-related workarounds for high-volume Domain Controllers?
4. Are there any specific Universal Forwarder versions or configuration recommendations that helped stabilize Security Event Log collection?

Any suggestions or similar experiences would be appreciated.

Thanks.

[PickleRick]

This might or might not be connected to your problem but I recall that windows eventlog inputs used to hit a performance limit at some point. A single input would get up to (in our case) around 12kEPS and wouldn't go pass that threshold. So it would start lagging behind during the day and would catch up during off-hours. No tweaking on UF's side (most importantly, adding pipelines) helped since the issue seemed to be with a single input, not UF as a whole - other inputs were processed OK. And - most importantly - it wasn't related to overall server performance since we had plenty of CPU left (and if I remember correctly, we weren't even hogging a single cpu up to 100%).

Luckily for us, we weren't ingesting locally generated events but those from Forwarded Events from WEF subscriptions. So after a bit of bending over backwards we created more event log channels and split the subscriptions across those channels so a single input wouldn't get overwhelmed.

If your underlying problem is the same, the input might be lagging up to a point where the eventlog file gets rotated on OS side and the event pointer is no longer valid. Raising eventlog file size could alleviate the problem a little by giving the input a bigger buffer to work with but of course wouldn't remediate it completely.

[frknklnc]

Thanks for sharing this. This sounds relevant to our case.

We are seeing similar behavior where the Universal Forwarder and splunk-winevtlog.exe remain running, other inputs may continue ingesting, but high-volume Windows Event Log channels such as Security stop progressing or fall behind.

We also observed significant delay on the Security channel. In one test window, the average delay was around 28 minutes, p95 was around 56 minutes, and max delay was close to 60 minutes. We are also seeing the following internal error:

WinEventLogChannel::queryEvtChannel: Unable to set seek position to the given bookmark

Your point about the input lagging behind until the Windows Event Log rolls/overwrites and the bookmark becoming invalid is interesting and may explain the bookmark behavior.

We will check the Security EPS rate, delay trend, and Windows Event Log retention/rollover behavior to see if the input is falling behind during high-volume periods.

In our case these are locally generated Domain Controller Security logs, not WEF Forwarded Events, so splitting WEF subscriptions is not directly applicable. But the single-channel throughput limitation theory is very useful.

Thanks again for the insight.

[frknklnc]

Quick update from our side.

We performed additional testing based on the high-volume / backlog theory.

On one affected Domain Controller, the Security Event Log volume was quite high. During one ingestion window, we observed approximately:

- max EPS: 1159
- average EPS: 436
- p95 EPS: 655

We also saw clear backlog behavior. The ingestion delay decreased from around 265 minutes to a few minutes while the forwarder was processing older Security events in burst mode.

On another affected Domain Controller, we increased the Windows Security Event Log maximum size to 8 GB to provide a larger OS-side event buffer. However, the Security log ingestion still stopped again after the change.

During that test window, the Security channel showed approximately:

- max EPS: 316
- average EPS: 234
- p95 EPS: 307
- max ingestion delay: around 21 minutes

So increasing the Windows Security Event Log buffer to 8 GB did not resolve the issue in our case.

We had also previously increased the Universal Forwarder throughput limit, but the issue still reproduced. Indexer receiving queues were also not blocked.

At this point, it seems the issue is not only related to UF output throughput or Windows Event Log retention size. It still looks like the WinEventLog input itself may stop progressing on high-volume Domain Controller Security logs.

We are continuing with Splunk Support, but any further suggestions around WinEventLog input behavior, bookmark handling, or high-volume Security log collection would be appreciated.

[PickleRick]

Well, 1kEPS is not that high actually. And did you just raise your uf thruput limit or remove it completely?

EDIT: There is also another test you could try but it might prove "politically challenging" in prod env situation - install additional different log forwarding solution (something that binds to event log and sends to syslog, for example) and see whether this piece of software keeps ingesti when UF stops. That would help definitely exclude issues with your windows box itself.

[frknklnc]

Thanks for the response.

You are right, ~1K EPS may not be extremely high in general. The reason we focused on EPS/backlog was because of the behavior we are seeing on the affected Domain Controllers.

To clarify the throughput part:

We increased the Universal Forwarder throughput limit, but we did not completely remove it. The value was increased during troubleshooting, for example up to 2048 KBps, but the issue still reproduced.

However, this does not look like a general UF-to-indexer output throughput issue because:

- The SplunkForwarder service remains running.
- splunk-winevtlog.exe remains running.
- Other inputs may continue to ingest.
- Indexer receiving queues were not blocked during the issue.
- We also tested an alternative receiving port, and the issue still reproduced.
- Increasing maxKBps did not prevent the issue.
- The issue affects specific Windows Event Log channels, mainly Security/System/Application.

The main behavior is that the Security Event Log input stops progressing after some time. Restarting the Universal Forwarder temporarily restores ingestion, but the issue happens again later.

We are seeing the following internal error repeatedly:

WinEventLogChannel::queryEvtChannel: Unable to set seek position to the given bookmark

We also tested increasing the Windows Security Event Log maximum size to 8 GB on another affected Domain Controller to provide a larger OS-side event buffer, but Security ingestion still stopped again after the change.

In that 8 GB test window, the Security channel had approximately:

- max EPS: 316
- average EPS: 234
- p95 EPS: 307
- max ingestion delay: around 21 minutes

So at this point, it does not seem to be only a UF output throughput limit or Windows Event Log buffer size issue.

The part we are trying to understand is why the WinEventLog input stops progressing while the UF service and splunk-winevtlog.exe are still running.

Would you recommend completely removing the UF throughput limit for a pilot test, or setting it to 0, to fully rule out throttling?

Also, are there any known settings or workarounds for WinEventLog bookmark handling or high-volume Domain Controller Security logs when the input process remains alive but the channel stops ingesting?

Any suggestions would be appreciated.

[PickleRick]

I can't "recommend" anything here but if I were debugging this kind of problem in my environment I'd try removing the thruput limit to rule out local choking.

[frknklnc]

Thanks, that makes sense.

We increased the UF throughput limit significantly during troubleshooting (up to 2048 KBps), but we did not completely remove the limit or set it to unlimited.

Since the issue still reproduced after increasing the limit, we initially assumed output throttling was probably not the primary cause. However, you make a good point that completely removing the limit would eliminate any possibility of local throttling.

We'll probably try this on a single affected Domain Controller as a controlled pilot test rather than across all systems.

Just to clarify, when you say removing the throughput limit, are you referring to setting:

[thruput]
maxKBps = 0

or do you recommend removing the parameter entirely from limits.conf?

Have you ever seen WinEventLog inputs stall like this because of local throughput throttling while splunk-winevtlog.exe and splunkd remained running?

[PickleRick]

You might still encounter issues if you overwhelm your indexers but it is relatively unlikely (and would mean you have severely undersized setup).

Removing the limit means setting the entry to zero. If you remove it, the setting from system/default will be effective so you want to explicitly set it to zero.