topic Re: How to resolve high increase in forwarders reporting errors, including "Failed to checkpoint" for channel='security'? in Splunk Search

How to resolve high increase in forwarders reporting errors, including "Failed to checkpoint" for channel='security'?

jcleary47 — Mon, 08 Jun 2020 17:59:25 GMT

We discovered that in early April, around the 7th, we had a HUGE increase in forwarders reporting this error:

ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe"" splunk-winevtlog - WinEventLogChannel::init: Failed to checkpoint for channel='security'

It's showing up on Windows 7, Windows 10, Server 2012, Server 2016 machines
Most of our forwarders are 7.1.2 but we have a few older versions out there.
Most of the hosts I've found with this error are Windows 10
No consistency in terms of OS or Forwarder versions where this error is showing up

Also, come to find out, this appears to correlate with events being missed from the Windows Event Security Logs and not being forwarded properly.

The ONLY thing that happened around when this seemed to spread like wildfire was Windows patches were deployed in our environment.

Has anyone else seen this issue?

The search we run to identify them is this if anyone wants to take a look to see if they have the errors as well:

index=_internal sourcetype=splunkd AND ERROR AND Security AND "Failed to checkpoint" AND log_level=ERROR 
| stats count by host

We have a case open with Splunk, but the suggested fixes do not seem to be resolving the errors.

Re: How to resolve high increase in forwarders reporting errors, including "Failed to checkpoint" for channel='security'?

splunk68 — Thu, 16 Apr 2020 08:52:21 GMT

Getting the same ERROR messages randomly on Windows 2016 servers with forwarders on 7.3.3.

Re: How to resolve high increase in forwarders reporting errors, including "Failed to checkpoint" for channel='security'?

morethanyell — Wed, 03 Jun 2020 18:04:42 GMT

No update?

Re: How to resolve high increase in forwarders reporting errors, including "Failed to checkpoint" for channel=

vanvan — Mon, 26 Oct 2020 19:19:51 GMT

I am having the same issue repeatedly in our Win'16 environment. Anyone found the cure yet?

Re: How to resolve high increase in forwarders reporting errors, including "Failed to checkpoint" for channel=

CarsonZa — Mon, 09 Nov 2020 23:59:21 GMT

converting UF to use service account seems to have fixed it for me. So if that's not possible at least you know that its a permissions issue with localsystem.