We occasionally come across servers with universal forwarders or devices that send syslog and their NTP service is off, which obviously causes some incorrect times to flow into Splunk.
How are other teams deal with it, detecting the issue? fixing, etc.
A handy definition of NTP ; -)
-- The Network Time Protocol (NTP) is a networking protocol for clock synchronization between computer systems over packet-switched, variable-latency data networks. In operation since before 1985, NTP is one of the oldest Internet protocols in current use. NTP was designed by David L. Mills of the University of Delaware
You could check out this NTPDrifter app to help identify the problem servers.
As for fixing the issue, I believe the best way is to correct the NTP problem at the source. Trying to manipulate your searches to account for it would really just be a band-aid and not even sure there would be a good way to do it.
Does it require a Splunk restart after NTP settings on the server is changed, if the server is a Heavy Forwarder ?