Extreme Latency with Windows Events on one Windows Event Collector. How do I troubleshoot?

[davidwaugh]

Hello i have two windows event collectors. 3 domain controllers send their events to one event collector (WEC01), and three send their events to another event collector.(WEC02)

From 8.00 onwards (eg the start of the working day) the events from WEC02 are getting progressively delayed up to about 20,000 seconds behind, before eventually catching up by about 4AM in the morning.

Both systems have the same configurations on them, which are managed by a deployment server.

I have looked at:
https://answers.splunk.com/answers/224727/why-is-my-universal-forwarder-showing-extreme-lag.html?utm...

And various other posts and have the following set:

limits.conf

[thruput]
maxKBps = 0

Outputs.conf

There doesnt appear to be any blockage in terms of indexer queues as other events are indexed fine and there is no latency. CPU, Memory and Network is all fine on the virtual machine. I can see no obvious reason why there is a delay.

Both Windows Event collectors are virtual machines. They may be on different physical hosts. There is a difference in latency in packets between the two hosts.

Here is a screenshot from the resouce monitor, network activity.

Slow Windows Event Collector (High Latency)

Fast Windows Event Collector (low latency)

[itrimble1]

Any difference in the configuration of WEC02 from a collector or UF configuration?
Is the volume of Events the same for WEC02?

[davidwaugh]

Nope, if anything there are fewer events on WEC02 than there are on WEC01.

[itrimble1]

Have you checked your indexers for congestion ? Have you checked the parsingQueue or the indexQueue ?

[davidwaugh]

Yep no congestion on the indexers. For instance at the same time I am ingesting syslog events and the delay for these is only a few seconds.

As far as I'm ware if this was an indexer problem, then all indexes would should as being behind, and not just one index, and not from only one forwarder.

[itrimble1]

How are is your ForwardedEvents Stanza configured?

[WinEventLog://ForwardedEvents]
sourcetype = WinEventLog:ForwardedEvents
disabled = 0
#start_from = oldest
current_only = 1
evt_resolve_ad_obj = 1
checkpointInterval = 5

Have you tried to change start_from to newest, restart, then switch it back to oldest ?