I want to set up an alert for when we start receiving events that are > 30 minutes off the idxtime. This would indicate a possible issue with TZ offset (perhaps the server admin changed the TZ without informing the Splunk admin.)
Anyone have such a search available?
I use the following query:
index=* | eval lag=_time-_indextime | search lag>1000 OR lag<-1000 | convert ctime(_indextime) as idxtime | stats latest(_time) , latest(idxtime) , max(lag) as lag by index,host,source | rangemap field=lag EST="9000-10000" MST="2000-4000" CST="5000-7000" PST="-1000-1000" | rename range as TZ
I've approximated the ranges but it gets it close enough. This is more to catch hosts sending events in their local time without a TZ specified but it can also catch hosts that are not NTP-synced.
View solution in original post