How do I determine if hosts are in the wrong timezone?

[Jason]

I have finished bringing in any archive data that I need to. Now all data flowing into Splunk is be up-to-the-minute, whether it be from a forwarder, a network port, or a script.

How do I tell if certain hosts sending data to Splunk have incorrect timezones?

[Jason · Accepted solution]

This is a very important question, as normalization of time is essential so you can have an accurate picture of what actually happened at a certain point in time.

(If you have a datasource in Central time, but your indexer is in Eastern and wasn't told otherwise, it will index the data as Eastern and events will show in Splunk as an hour behind other events that happened that minute!)

One method is to try running this search. Run it over a period that would capture at least one event from each host, such as a day. It compares _time (the epoch time version of the time extracted from the event, adjusted with any timezone in the event or in props.conf) to _indextime (the epoch time version of when Splunk indexed the event). If your data is all current, these should be only seconds off if a datasource is properly configured.

* | dedup host | eval timediff=(_time-_indextime) | eval HoursOff=round(timediff/3600) | search NOT HoursOff=0 | rename _indextime as IndexTime | eval Time=_time | table Time, IndexTime, HoursOff, host

Remember this checks Index Time against Time extracted from the event. It assumes the most recent event from each host is coming in at current time, and isn't archive data.

I'm going to get this into an App, along with a script to add the timezone into the source field for troublesome linux servers. At some point.