We have Splunk Enterprise 7.2.6 in our environment. I noticed there are latencies (difference between _time and _indextime from 1hr to 10hrs). My Splunk Heavy Forwarders are in GMT timezone, hence I have set TZ = UTC for few of the sourcetypes in props.conf of HF and it worked.
Still I am seeing time difference of 5hrs to 10hrs on few hosts for specific sourcetypes. I am unsure which is creating latency in _indextime. Attached screenshot for reference.
Can someone please assist me to fix this issue?
Have you determined if that is a time extraction/time zone issue or a true lag of indexing?
Where the data that has this lag is coming from? Is that windows/linux logs or syslog? Or something else?
I can see data is coming from syslog. Below are few of the sourcetypes which also has index lagging
I also noticed data_zone associated with these hosts are either "local" or "0".
Usually if it is lag caused by some data collection issue, it will fluctuate. Any offsets that are pretty static and also very close to an exact nr of hours offset is likely a timezone config issue.
One way to be sure is to check on the affected host when the event really happened (assuming the host also stores the events locally for a while). Or trigger some specific event on the host and see when that becomes visible in Splunk and with what timestamp.
I would start by checking the timezone config of the originating hosts. Especially if the offset is clearly an (almost) exact nr. of hours.
Quite possibly a US host is sending data with timestamps in its local timezone (US Central, GMT-5) to your HF / Indexer(s) which you configured to interpret the data as GMT, causing this skew.
Ideally source systems are configured to report timezone in the timestamp, to prevent any confusion (and avoid specific config to adjust for offsets). If that is not an option, another solution can be to install a Heavy/Universal Forwarder in the same timezone as the initial collection point. If that is also not an option, you'll need to put specific config in place to assign the correct timezone (e.g. using host based stanzas in props.conf).
Source data is coming from timezone EST, hence I have manually added TZ=EST (also tried GMT-5) definiton for sepcific host on HF. Both are failed 😞
I am still seeing latency in indextime. As I mentioned earlier this is syslog data. Is there any other way I can fix this issue?
No, you cannot combine host and sourcetype criteria like that. It is either one or the other.
As for using 'EST', not sure if that is ideal. It is listed as deprecated:
This wiki page is referred in Splunk docs as the reference for acceptable time zone names, see: https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Applytimezoneoffsetstotimestamps
Also: if your source system is using DST during summer, better use for example
Also: make sure to deploy this config on the first HF that touches the data. And make sure the host field is actually populated with the value used in props.conf. Note that for syslog data splunk TAs often override the host value based on event content, so what you see in splunk web may not be the host name with which the data arrives at the HF originally.
I have uploaded few more snapshots to this query. I am unsure if you have ever faced this issue. Could you please review and provide your suggestions on what can be done?