Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
×
Join the Conversation
Without signing in, you're just watching from the sidelines. Sign in or Register to connect, share, and be part of the Splunk Community.
- Find Answers
- :
- About frknklnc
frknklnc
Observer
Member since:
07-21-2026
07-23-2026
Community Statistics
| Posts | 6 |
| Solutions | 0 |
| Karma Given | 0 |
| Karma Received | 0 |
| Member Since | 07-21-2026 |
Activity Feed
- Posted Re: Windows Event Log collection stops on Universal Forwarder while splunk-winevtlog.exe keeps running on Splunk Search. 07-23-2026 01:17 PM
- Posted Re: Windows Event Log collection stops on Universal Forwarder while splunk-winevtlog.exe keeps running on Splunk Search. 07-23-2026 12:39 PM
- Posted Re: Windows Event Log collection stops on Universal Forwarder while splunk-winevtlog.exe keeps running on Splunk Search. 07-23-2026 10:52 AM
- Posted Re: Windows Event Log collection stops on Universal Forwarder while splunk-winevtlog.exe keeps running on Splunk Search. 07-23-2026 10:52 AM
- Posted Re: Windows Event Log collection stops on Universal Forwarder while splunk-winevtlog.exe keeps running on Splunk Search. 07-21-2026 02:56 PM
- Posted Windows Event Log collection stops on Universal Forwarder while splunk-winevtlog.exe keeps running on Splunk Search. 07-21-2026 06:30 AM
- Tagged Windows Event Log collection stops on Universal Forwarder while splunk-winevtlog.exe keeps running on Splunk Search. 07-21-2026 06:30 AM
- Tagged Windows Event Log collection stops on Universal Forwarder while splunk-winevtlog.exe keeps running on Splunk Search. 07-21-2026 06:30 AM
- Tagged Windows Event Log collection stops on Universal Forwarder while splunk-winevtlog.exe keeps running on Splunk Search. 07-21-2026 06:30 AM
- Tagged Windows Event Log collection stops on Universal Forwarder while splunk-winevtlog.exe keeps running on Splunk Search. 07-21-2026 06:30 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 |
07-23-2026
01:17 PM
Thanks, that makes sense. We increased the UF throughput limit significantly during troubleshooting (up to 2048 KBps), but we did not completely remove the limit or set it to unlimited. Since the issue still reproduced after increasing the limit, we initially assumed output throttling was probably not the primary cause. However, you make a good point that completely removing the limit would eliminate any possibility of local throttling. We'll probably try this on a single affected Domain Controller as a controlled pilot test rather than across all systems. Just to clarify, when you say removing the throughput limit, are you referring to setting: [thruput] maxKBps = 0 or do you recommend removing the parameter entirely from limits.conf? Have you ever seen WinEventLog inputs stall like this because of local throughput throttling while splunk-winevtlog.exe and splunkd remained running?
... View more
07-23-2026
12:39 PM
Thanks for the response. You are right, ~1K EPS may not be extremely high in general. The reason we focused on EPS/backlog was because of the behavior we are seeing on the affected Domain Controllers. To clarify the throughput part: We increased the Universal Forwarder throughput limit, but we did not completely remove it. The value was increased during troubleshooting, for example up to 2048 KBps, but the issue still reproduced. However, this does not look like a general UF-to-indexer output throughput issue because: - The SplunkForwarder service remains running. - splunk-winevtlog.exe remains running. - Other inputs may continue to ingest. - Indexer receiving queues were not blocked during the issue. - We also tested an alternative receiving port, and the issue still reproduced. - Increasing maxKBps did not prevent the issue. - The issue affects specific Windows Event Log channels, mainly Security/System/Application. The main behavior is that the Security Event Log input stops progressing after some time. Restarting the Universal Forwarder temporarily restores ingestion, but the issue happens again later. We are seeing the following internal error repeatedly: WinEventLogChannel::queryEvtChannel: Unable to set seek position to the given bookmark We also tested increasing the Windows Security Event Log maximum size to 8 GB on another affected Domain Controller to provide a larger OS-side event buffer, but Security ingestion still stopped again after the change. In that 8 GB test window, the Security channel had approximately: - max EPS: 316 - average EPS: 234 - p95 EPS: 307 - max ingestion delay: around 21 minutes So at this point, it does not seem to be only a UF output throughput limit or Windows Event Log buffer size issue. The part we are trying to understand is why the WinEventLog input stops progressing while the UF service and splunk-winevtlog.exe are still running. Would you recommend completely removing the UF throughput limit for a pilot test, or setting it to 0, to fully rule out throttling? Also, are there any known settings or workarounds for WinEventLog bookmark handling or high-volume Domain Controller Security logs when the input process remains alive but the channel stops ingesting? Any suggestions would be appreciated.
... View more
07-23-2026
10:52 AM
Quick update from our side. We performed additional testing based on the high-volume / backlog theory. On one affected Domain Controller, the Security Event Log volume was quite high. During one ingestion window, we observed approximately: - max EPS: 1159 - average EPS: 436 - p95 EPS: 655 We also saw clear backlog behavior. The ingestion delay decreased from around 265 minutes to a few minutes while the forwarder was processing older Security events in burst mode. On another affected Domain Controller, we increased the Windows Security Event Log maximum size to 8 GB to provide a larger OS-side event buffer. However, the Security log ingestion still stopped again after the change. During that test window, the Security channel showed approximately: - max EPS: 316 - average EPS: 234 - p95 EPS: 307 - max ingestion delay: around 21 minutes So increasing the Windows Security Event Log buffer to 8 GB did not resolve the issue in our case. We had also previously increased the Universal Forwarder throughput limit, but the issue still reproduced. Indexer receiving queues were also not blocked. At this point, it seems the issue is not only related to UF output throughput or Windows Event Log retention size. It still looks like the WinEventLog input itself may stop progressing on high-volume Domain Controller Security logs. We are continuing with Splunk Support, but any further suggestions around WinEventLog input behavior, bookmark handling, or high-volume Security log collection would be appreciated.
... View more
07-23-2026
10:52 AM
Quick update from our side. We performed additional testing based on the high-volume / backlog theory. On one affected Domain Controller, the Security Event Log volume was quite high. During one ingestion window, we observed approximately: - max EPS: 1159 - average EPS: 436 - p95 EPS: 655 We also saw clear backlog behavior. The ingestion delay decreased from around 265 minutes to a few minutes while the forwarder was processing older Security events in burst mode. On another affected Domain Controller, we increased the Windows Security Event Log maximum size to 8 GB to provide a larger OS-side event buffer. However, the Security log ingestion still stopped again after the change. During that test window, the Security channel showed approximately: - max EPS: 316 - average EPS: 234 - p95 EPS: 307 - max ingestion delay: around 21 minutes So increasing the Windows Security Event Log buffer to 8 GB did not resolve the issue in our case. We had also previously increased the Universal Forwarder throughput limit, but the issue still reproduced. Indexer receiving queues were also not blocked. At this point, it seems the issue is not only related to UF output throughput or Windows Event Log retention size. It still looks like the WinEventLog input itself may stop progressing on high-volume Domain Controller Security logs. We are continuing with Splunk Support, but any further suggestions around WinEventLog input behavior, bookmark handling, or high-volume Security log collection would be appreciated.
... View more
07-21-2026
02:56 PM
Thanks for sharing this. This sounds relevant to our case. We are seeing similar behavior where the Universal Forwarder and splunk-winevtlog.exe remain running, other inputs may continue ingesting, but high-volume Windows Event Log channels such as Security stop progressing or fall behind. We also observed significant delay on the Security channel. In one test window, the average delay was around 28 minutes, p95 was around 56 minutes, and max delay was close to 60 minutes. We are also seeing the following internal error: WinEventLogChannel::queryEvtChannel: Unable to set seek position to the given bookmark Your point about the input lagging behind until the Windows Event Log rolls/overwrites and the bookmark becoming invalid is interesting and may explain the bookmark behavior. We will check the Security EPS rate, delay trend, and Windows Event Log retention/rollover behavior to see if the input is falling behind during high-volume periods. In our case these are locally generated Domain Controller Security logs, not WEF Forwarded Events, so splitting WEF subscriptions is not directly applicable. But the single-channel throughput limitation theory is very useful. Thanks again for the insight.
... View more
07-21-2026
06:30 AM
Hello everyone, We are investigating an intermittent Windows Event Log ingestion issue on multiple Windows Domain Controllers. Affected channels: - WinEventLog:Security - WinEventLog:System - WinEventLog:Application Observed behavior: - Windows Event Viewer continues to generate new events. - The SplunkForwarder service remains running. - The splunk-winevtlog.exe process remains running. - Other inputs such as DNS Server and Directory Service may continue ingesting. - Security, System, and Application ingestion stops after some time. - Restarting the SplunkForwarder service temporarily restores ingestion, but the issue occurs again later. Environment: - Splunk Enterprise version: 9.4.12 - Universal Forwarder versions tested: 9.4.12, 10.2.4, and 10.4.0 - Windows Domain Controller servers - Indexer Cluster environment Important internal log message observed: WinEventLogChannel::queryEvtChannel: Unable to set seek position to the given bookmark Troubleshooting already performed: - Verified that Windows Event Viewer continues generating events. - Verified that the SplunkForwarder service and splunk-winevtlog.exe remain running. - Tested Universal Forwarder versions 9.4.12, 10.2.4, and 10.4.0. - Tested current_only = 1. - Tested evt_resolve_ad_obj = 0. - Cleared WinEventLog checkpoint and persistent storage. - Increased Universal Forwarder maxKBps. - Verified that indexer TCP input queues are not blocked. - Tested an alternative Splunk receiving port instead of 9997. - Confirmed that the SplunkForwarder service runs as Local System. - Confirmed that EDR/antivirus is not blocking splunkd.exe or splunk-winevtlog.exe. After upgrading to Universal Forwarder 10.4.0, the Security channel initially processed backlog events in burst mode, but later ingestion stopped again. We are also working with Splunk Support and have provided DEBUG diagnostic logs. Questions: 1. Has anyone experienced Windows Event Log channels stopping while splunk-winevtlog.exe remains running? 2. Did changing start_from, current_only, or checkpoint behavior resolve it in your environment? 3. Are there any known Windows Event Log, bookmark, or subscription-related workarounds for high-volume Domain Controllers? 4. Are there any specific Universal Forwarder versions or configuration recommendations that helped stabilize Security Event Log collection? Any suggestions or similar experiences would be appreciated. Thanks.
... View more
Labels
- Labels:
-
Other
Contact Me
| Online Status |
Offline
|
| Date Last Visited |
07-23-2026
01:17 PM
|
