topic Re: WEF - Universal Forwarder multiple Windows TA's / performance issues in Getting Data In

WEF - Universal Forwarder multiple Windows TA's / performance issues

ljo4497 — Tue, 05 Nov 2024 00:04:11 GMT

Hi,

We currently have a centralized WEF collection server that collects all windows logs across the environment.
This includes forwarding sysmon,application,system channels etc... to the collector.

Everything ends up in ForwardedEvents on the WEF collection server. I've installed a UF on this host.

I have the windows TA deployed with the following input stanza

#[WinEventLog://ForwardedEvents] #disabled = 0 #index = wef #start_from = oldest #current_only = 0 #batch_size = 50 #checkpointInterval = 15 #renderXml=true #host=WinEventLogForwardHost

I have 2 problems currently.

The splunk universal forwarder doesn't appear to be keeping up with the number of windows event logs coming to the WEF collector. ~1000 hosts. Another (different) SIEM collector for WEF keeps up fine on the same host and collects all logs. i'm able to compare what one collector is collecting vs the Splunk UF. I've tried adjusting the batch_size and checkpoint interval as above.

I want to split certain windows channels in the ForwardedEvents channel to different indexes. I have tried deploying the microsoft sysmon TA and adding a new input with the following configuration.

#[WinEventLog://ForwardedEvents] #disabled = true #index = wef-sysmon #start_from = oldest #current_only = 0 #batch_size = 50 #checkpointInterval = 15 #renderXml=true #host=WinEventLogForwardHost #whitelist = $XmlRegex='Microsoft-Windows-Sysmon'

i then add

blacklist = $XmlRegex='Microsoft-Windows-Sysmon'

to the windows TA.

Then everything seems to stop. I stop receiving all events on my indexer.

I've also tried adding multiple inputs with differing indexes and whitelist/blacklists in the windows TA to no avail.

Would someone be able to point me in the right direction?

Re: WEF - Universal Forwarder multiple Windows TA's / performance issues

PickleRick — Tue, 05 Nov 2024 07:02:01 GMT

Wineventlog inputs have been known for having performance problems above certain EPS threshold. It usually doesn't manifest itself in local events ingestion but shows when pulling WEF-ed logs. Adding additional pipelines doesn't help.

The way around it (other than setting up more WEC hosts and splitting WEF subscriptions among them is to create more eventlog channels and split your subscription into several channels. The performance problems for eventlog inputs seem to be at single input level so if you're getting stuck around 10k EPS with single input you should be able to get up to 40k EPS if you split your ForwardedLogs into 4 channels.

Unfortunately, it's a bit of work to set it up and you need to create custom dll for that.

https://learn.microsoft.com/en-gb/archive/blogs/russellt/creating-custom-windows-event-forwarding-logs

https://github.com/palantir/windows-event-forwarding/blob/master/windows-event-channels/README.md

Re: WEF - Universal Forwarder multiple Windows TA's / performance issues

ljo4497 — Tue, 05 Nov 2024 22:00:48 GMT

Thanks @PickleRick

This was my thinking as well. We're really only doing around 1500 EPS roughly. So unsure why some messages are not making it through and others aren't.

Yeah i've looked into the links you've provided previously. the problem is getting a hold of ecmangen.exe as you have to install quite an old Win 10 SDK to access it as it's been removed from all recent SDK's.

We're running server 2022 on our WEF Collector.

Re: WEF - Universal Forwarder multiple Windows TA's / performance issues

MuS — Tue, 05 Nov 2024 23:39:09 GMT

Hi there,

Missing events from WEF/WEC can be cause by the file size, if too small they rotate away before the UF even has a change to read it .. don't ask how I know 😉

Increasing the size for the forwardedevents channel will help resolving this.

Hope this helps ...

cheers, MuS

Re: WEF - Universal Forwarder multiple Windows TA's / performance issues

PickleRick — Wed, 06 Nov 2024 10:23:22 GMT

Yup. If you start lagging behind (in our case we were about 2-2.5 hours behind during midday; we would catch up during evening-night) and Windows decides to rotate the log file, you'll end up missing events probably.

Re: WEF - Universal Forwarder multiple Windows TA's / performance issues

ljo4497 — Wed, 13 Nov 2024 00:27:16 GMT

Thanks all,

I've split out the Forwarded events and subscriptions to be more granular. And the dedicated sysmon channel + the TA is working well.

I think we're roughly running 9 minutes behind. which isn't too bad, but i want to ensure we don't miss any logs. I'm still collecting some event IDs, but not seeing them in Splunk at all. I am seeing them in other solutions.

Can i increase the cache size of the universal forwarder itself?
I've increased the persistentCacheSize to 10GB, but unsure if i've set this property correctly or if it impacts the windows_TA

Thanks

Re: WEF - Universal Forwarder multiple Windows TA's / performance issues

ljo4497 — Wed, 13 Nov 2024 02:31:42 GMT

When i enable

[WinEventLog]
persistentQueueSize=5GB

in the windows_ta, all event flow stops.

I see the queue file created in var/run/splunk/exec

but no events are indexed. I remove that stanza, and events flow again...

Re: WEF - Universal Forwarder multiple Windows TA's / performance issues

Kamachi — Tue, 14 Apr 2026 11:20:29 GMT

same here, did u manage to resolve the issue?

Re: WEF - Universal Forwarder multiple Windows TA's / performance issues

inventsekar — Wed, 15 Apr 2026 10:04:11 GMT

Hi @Kamachi

as this is a 2 years old topic, may i ask you, what steps you have performed so far, some more details like splunk cloud or onprim, etc

Re: WEF - Universal Forwarder multiple Windows TA's / performance issues

Kamachi — Wed, 15 Apr 2026 12:21:39 GMT

Hi, there, fixed it already.

- on prem
- WEC server with UF aggregating logs from a lot of win servers

Applying the following inputs.conf caused the data flow from the WEC server to stop:

[WinEventLog://ForwardedEvents] disabled = 0 start_from = oldest current_only = 0 checkpointInterval = 5 index = wineventlog [WinEventLog] persistentQueueSize = 30GB

The Monitoring Console still showed data being sent to the indexer
_internal logs (splunkd) confirmed that data was being forwarded
However, no data appeared in the target index
Removing the [WinEventLog] stanza restored data flow
This suggested a configuration merge issue, where the index setting was lost or misconfigured during processing, resulting in data being dropped at the indexer level

Explicitly defining the target index in the [WinEventLog] stanza resolved the issue. Data is now correctly buffered using the persistent queue and indexed as expected:

[WinEventLog://ForwardedEvents] disabled = 0 start_from = oldest current_only = 0 checkpointInterval = 5 index = wineventlog [WinEventLog] persistentQueueSize = 30GB index = wineventlog

Anyway thanks for the feedback!
Best regards.