I really cannot follow what you have said well enough to give you a full answer but I can tell you a few things that will allow you to help yourself. The main search you need is this one (assuming that your forwarder pipeline has near-zero latency, which is usually the case):
index = * | eval lagSecs = _indextime - _time | stats avg(lagSecs) by index,host,sourcetype
The avg should NEVER be negative and should generally be in the low-hundreds or smaller. Assuming you have correct clock time and no drift (NTP in place), this search will show you which hosts for which sourcetypes need the TZ value to be adjusted but there is another thing to consider: if you have overridden your host or sourcetype, you must use the original/non-overridden value for your stanza header. Many times, this is not practical and the best (only?) other option is to have each host write to his own private subdirectory and then use a source -based stanza (which is now an analog for host ) in props.conf to set the TZ. If you are using DST, make sure that you do NOT use the GMT* -based TZs but use the US/* -based TZs.
... View more