All of a sudden my 4.0.9 Splunk server is no longer forwarding the WinEventLog:Security logs onto my 4.1.4 Linux indexer. This was working fine and I can see the communication is established between the 2 systems. What's at issue?
This is a known defect, SPL-31339: WinEventLog:Security logs stop getting indexed and Splunkd.log displays the following errors:
ERROR WinEventLogChannel - initOld: Failed to initialize checkpoint for Windows Event Log channel 'Security'
ERROR WinEventLogInputProcessor - main-thread: Failed to initialize Windows Event Log 'Security'
This is likely due the Windows host not shutting down properly and the %SPLUNK_HOME%\var\lib\splunk\persistentstorage\WinEventLog\Security_checkpoint file is empty (size =1KB)
The workaround is to shutdown Splunk on the Windows host, remove the file and restart Splunk to create a new Security_checkpoint file. This will allow the security logs to start indexing again.