I have a Splunk Forwarder running on Windows 2012 and I'm monitoring a share with archived .evtx files from other Windows servers. I discovered that Splunk was ingesting most small event logs (less than 1092 KB) but skipped larger files with events numbering in the thousands. While examining the _internal index within Splunk Enterprise, it was seeing and processing the large files, however, the total events always equaled zero. If I opened the .evtx file on a Windows computer and exported 256 events, the max it allowed me to export at a time, to a new .evtx file, Splunk would see, process the right number of events and ingest the file without any issue.
Update: Further troubleshooting has revealed that the Windows security event log is the only one with this problem. Application and system logs ingest with events numbering in the thousands with no problem. I have a workaround in place to export security logs every 5 minutes. Since this keeps events to around 500 or less per file, Splunk processes the files with the correct number of events.
Solution: Working with Splunk support, it was determined that this issue was related to a bug in the version 6.5.1 Universal Forwarder. It has been fixed in 6.5.2.
Solution: Working with Splunk support, it was determined that this issue was related to a bug in the version 6.5.1 Universal Forwarder. It has been fixed in 6.5.2.