Inconsistent Data being sent to indexer

[david_balch]

I have Splunk setup on an air gapped network (no internet connection). The search head is a single instance running 8.1.1. There are about 320 machines on the network. It's mostly Windows 10 clients, the servers are 2016 & 2019. The clients are running the universal forwarder 7.1.1. I've created a couple of reports to see different things about the computers on the network (OS, if the Print log is turned on, etc). I get very inconsistent results from these reports (and other searches I'll do). E.G. - If I do a search on the Security event log for EventCode 4608 (restart event) I'll only get about 80 results (clients reboot nightly) when I should be getting closer to about 300. I've searched the event logs of machines that aren't on the report and they have the event code logged, but it's not being reported to splunk. I've checked everything I can think of, uninstalled/reinstalled the forwarder, installed different versions of the forwarder, etc. I can't figure out why one machine will report all events and another only reports some events (4624, 4627, 4634). Has anyone else had this issue?  Thank you 

[david_balch]

Unfortunately that did not fix the issue. 

[david_balch]

Thank you for the reply. 
Splunk TA is 8.1.1-R62a79cd.

I'll try what you suggested and see how it works. 

[m_pham]

What version of the Splunk_TA_windows are you using? Is it installed on all of your servers (UF/IDXC/SH/SHC)?

I'm not sure if this is related to your issue:

2015-04-14

SPL-99687, SPL-129637

Splunk universal forwarder is 7-10 days behind recent Windows Security and system log events. Workaround: To mitigate this, edit the following stanza in inputs.conf: [WinEventLog://Security] evt_resolve_ad_obj = 0.

Link: https://docs.splunk.com/Documentation/Splunk/8.1.1/ReleaseNotes/KnownIssues#Windows-specific_issues