Missing events from search for specific hosts running UF

[JeremyHagan]

I have around 80 identically configured branch office domain controllers. They all get their config from the deployment server which defines a few file monitors and Windows event logs.

The config works on the majority of DC's but on two of them I can't see the WinEventLog:Security events. I can see events from other flat-file sources such as DNS server log files and the Active Directory sourcetype is also returning events.

If I check the license usage of that host, I can see that data from that sourcetype is being logged as used. So I suspect that the UF is sending the data and that the indexer is receiving it, but it is just not showing up in search.

Any ideas?

[s2_splunk]

Possible timestamp extraction issues resulting in timestamps in the future for the affected hosts and sourcetypes?
It is weird that it would only affect your Security event log.

I would start by checking your _internal index for error messages logged by splunkd in the DateParserVerbose category:

 index=_internal sourcetype=splunkd component=DateParserVerbose host=yourMissingHost

and see if anything shows up with a message text of

A possible timestamp match (dow mon dd HH:MM:SS YYYY) is outside of the acceptable time window.

or similar (assuming you are forwarding splunkd logs from forwarders.

[JeremyHagan]

Hi,

Thanks for the reply. I should mention that I've done some "All Time" searches against this host in case the events were showing up in the future with no luck. As you say, being a DC, I'd have other problems with time sync. The server is definitely in a different time zone, but I have two servers at the site and it is only the DC that is not forwarding Windows Event logs and they are both covered by the same entry in the Splunk config for time zone adjustment.

We are forwarding Splunkd logs and I checked for DateParserVerbose errors but nothing came up. In fact the only ERROR present is one about it not being able to locate the PDC emulator, but every DC has that error.