Search to discover events indexed with incorrect TZ offset?

[the_wolverine]

I want to set up an alert for when we start receiving events that are > 30 minutes off the idxtime. This would indicate a possible issue with TZ offset (perhaps the server admin changed the TZ without informing the Splunk admin.)

Anyone have such a search available?

[the_wolverine · Accepted solution]

I use the following query:

index=* | eval lag=_time-_indextime | search lag>1000 OR lag<-1000 | convert ctime(_indextime) as idxtime |  stats latest(_time) , latest(idxtime) , max(lag) as lag by index,host,source | rangemap field=lag EST="9000-10000" MST="2000-4000" CST="5000-7000" PST="-1000-1000" | rename range as TZ

I've approximated the ranges but it gets it close enough. This is more to catch hosts sending events in their local time without a TZ specified but it can also catch hosts that are not NTP-synced.