Why WinEventLog collection on busy domain controllers were slow before, and are now fast on Splunk 6.0

[mataharry]

In splunk 4.* and 5.* I used to have Splunk Universal forwarders on Windows Domain Controllers.

The volume of events on the channels : Security, Application, System etc... were high during the daytime, and the forwarder couldn't keep up. Over the day, we were experiencing delay between the events timestamp and the _indextime (once indexed and searchable). And during the night the events were catching up.

You could see the behavior with a search like this for the previous day :
sourcetype=WinEventLog host=myhost | delay_sec=_indextime-_time | timechart avg(delay_sec) by sourcetype

But since I switched to Universal forwarders on 6.0, things started to get better.
What magic happened ?

PS : To boost speed :
- We already removed the default thruput limit, and were having unlimited speed in limits.conf
[thruput]
maxKBps=0
- and the AD object name resolver was the local AD.

[yannK · Accepted solution]

The magic is called Modular Inputs, introduced in Splunk 6.0
Most of the Windows inputs were rewritten to use them, and for the WinEventLog, they improved the performance.

see the modular inputs description http://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ModInputsIntro
and the WinEventLog
see http://docs.splunk.com/Documentation/Splunk/latest/Data/MonitorWindowsdata

The old input version were hitting performance degradation above many events per seconds, and the delay was proportional to the number of events.

• The collection was single threaded (and was switching context between every channels).
• And the object name resolution was also slower to open the connection to query the AD and load the libraries for every event.
• The only filtering possible was at parsing time (on the indexers) using nullQueue.

With modular inputs since 6.0, the performance improved because :

• The collection is multithreaded,
• the name resolution step keeps the libraries loaded in memory.
• a new whitelist/blacklist parameter allows you to filter events based on EventCode directly at the source.

I recommend to upgrade to 6.0 to benefit of this improvement.
Also if you cannot upgrade all your infrastructure, 6.0 forwarders are compatible with 5.0 indexers.