How do you troubleshoot missing windows event logs?

[ABasit10]

I have been noticing that some windows event logs are not appearing in the Splunk search. For example the event code for windows restart is 1074 but whenever I use the search string below, I do not get any results back within the specific time period. I know for a fact that the event codes are not blacklisted in the configuration files as I can view older logs for the same event code. When checking the logs in Event Viewer, I can also see the logs for the windows restart event. I am not sure why the event is logged sometimes but other times it isn't. This issue isn't exclusive to a particular windows event as I have noticed missing logs for other events as well.

Query: index=* host=DC* EventCode=1074

Help with troubleshooting the issue would be greatly appreciated.

[jeremyfer]

Did you ever find a solution to this @ABasit10 ?
We have the exact same forwarder config on hundreds of other machines, and it's functioning fine. On one machine Splunk seems to be skipping lots of events, I'm looking specifically for 7001 (login), and its never there, but also even long after a login many events are missing. I can see in event viewer events A,B,C we might only get event B into Splunk.

E.g. Events in Event Viewer, only the highlighted ones are coming through.

But we seem to be missing a large selection of Events.

Related Forwarder Config

[WinEventLog://System]
disabled = 0
start_from = oldest
current_only = 0
checkpointInterval = 10
index = wineventlog
renderXml=false

[to4kawa]

https://qiita.com/odorusatoshi/items/5a703b9befc253ab7deb

@jeremyfer 

this blog is japanese, but there is a lot of trouble shoot method.
please check this.

[nickhills]

Do you have current_only=1 set on your windows forwarders?

This setting results in ONLY events generated while Splunk is running being sent.
This means that if you shutdown your host, any logs generated between the time the Splunk process stops and restarts will never be indexed.

[ABasit10]

Thanks for the response. I have validated that current_only=0 is the value set on the forwarders. The strangest thing is that the event is sometimes logged and other time is isn't. For testing, i had manually restarted the host to see if the event code 1074 was being logged by Splunk. I wasn't able to find any event between for that time frame however, i saw a windows host restart event on a different day with a different time stamp.

I am not sure why some events for this particular event code (1074) are being logged and others aren't.

[dillardo_2]

Have you checked your buckets? If they are being frozen prematurely due to storage pressure, the data might have been deleted before retention policy expiry.