Find a misconfigured host time?

[b1nki3]

I'm seeing this in my splunkd.log:

07-09-2010 12:53:21.299 WARN DateParserVerbose - Time parsed (Fri Jul 9 12:53:18 2010) is too far away from the previous event's time (Fri Jul 9 18:52:37 2010) to be accepted. If this is a correct time, MAX_DIFF_SECS_AGO (3600) or MAX_DIFF_SECS_HENCE (604800) may be overly restrictive.

This seems to be due to a mismatch between local time and GMT. Splunk seems to be correcting this itself. Due to this fact, I'm not sure how I'd use splunk to find the offending host. Any ideas?

[gkanapathy]

You could just search for the literal time string "Fri Jul 9 12:53:18 2010" that is indicated by the log line. Hopefully there are just a few within that one-second span and any deviation from the Splunk-recorded timestamp will be obvious.

[Lowell]

Are you sure you don't have a device that is logging both local and UTC timezone events within the same file? I've seen really weird stuff like that happen before, sometimes with syslog events.

Have you tried looking for events that occur on the 2nd timestamps listed? You can search on the first timestamp because that event was dropped. (You could set MAX_DIFF_SECS_AGO=28800 to allow for an 8 hour window to help track down this issue.) Try entering the timestamp into the date range picker manually and see what all events you can find. You may also find other helpful events from DateParserVerbose that could help you track this down.