WinEventLog:Security - Unexpected Increase in Events

[jeremyarcher]

This isn't specifically a Splunk question but the effects of this have put my Splunk server into craziness.

On July 5th (late in the evening) ourl systems started generating a crazy number of AD Event Code 4624 events. Usually they would do around 10-15 per hour. Now they are doing 18-20k per hour.

Has anyone seen anything like this before? Our domain controllers (Win2012R2) were patched that day but no group policy changes.

Anyone else seen anything similar or a way to tune the number of these down?

[jeffland]

Assuming you are running a Universal Forwarder on the source of these logs, you could try the following in limits.conf:

[thruput]
maxKBps = <integer>
* If specified and not zero, this limits the speed through the thruput processor to the specified rate in kilobytes per second.
* To control the CPU load while indexing, use this to throttle the number of events this indexer processes to the rate (in KBps) you specify. 

Reducing this setting might help to throttle the number of events you receive. Actually, I am not sure how Splunk handles the remaining data; I would presume it just piles up in the buffer of the forwarder until that is full and then use the disk as buffer, just as the fowarder does with indexing acknowledgement enabled. The way I understood you, you want the overflowing events dropped, but I don't know how to influence this behavior.

If you want to figure out the root of this problem and in the meantime disregard all those events, you can simply route them to the nullqueue. See here for how that is done (your regex would then just contain 4624).

[jeremyarcher]

Thanks! This is helpful for keeping things under control until I can find the root cause.

[jeffland]

I have also just heard of this nice little solution in-between indexing none and all such events.