We discovered that in early April, around the 7th, we had a HUGE increase in forwarders reporting this error:
ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe"" splunk-winevtlog - WinEventLogChannel::init: Failed to checkpoint for channel='security'
Also, come to find out, this appears to correlate with events being missed from the Windows Event Security Logs and not being forwarded properly.
The ONLY thing that happened around when this seemed to spread like wildfire was Windows patches were deployed in our environment.
Has anyone else seen this issue?
The search we run to identify them is this if anyone wants to take a look to see if they have the errors as well:
index=_internal sourcetype=splunkd AND ERROR AND Security AND "Failed to checkpoint" AND log_level=ERROR | stats count by host
We have a case open with Splunk, but the suggested fixes do not seem to be resolving the errors.