In splunk 4.* and 5.* I used to have Splunk Universal forwarders on Windows Domain Controllers.
The volume of events on the channels : Security, Application, System etc... were high during the daytime, and the forwarder couldn't keep up. Over the day, we were experiencing delay between the events timestamp and the _indextime (once indexed and searchable). And during the night the events were catching up.
You could see the behavior with a search like this for the previous day : sourcetype=WinEventLog host=myhost | delay_sec=_indextime-_time | timechart avg(delay_sec) by sourcetype
But since I switched to Universal forwarders on 6.0, things started to get better. What magic happened ?
PS : To boost speed :
- We already removed the default thruput limit, and were having unlimited speed in limits.conf [thruput]
- and the AD object name resolver was the local AD.