We have Splunk Enterprise 7.2.6 in our environment. I noticed there are latencies (difference between _time and _indextime from 1hr to 10hrs). My Splunk Heavy Forwarders are in GMT timezone, hence I have set TZ = UTC for few of the sourcetypes in props.conf of HF and it worked.
Still I am seeing time difference of 5hrs to 10hrs on few hosts for specific sourcetypes. I am unsure which is creating latency in _indextime. Attached screenshot for reference.
Can someone please assist me to fix this issue?
Where is the data coming from exactly and at what point is it getting cooked ? If it's going via a HF make sure you set the TZ configuration for the
As per suggestions I have removed EST and used appropriate TZ definiton. I also tried UTC-05:00 for that speciic host, but it is not working 😞
Please find below screenshots
1. Difference in event timestamp and search time.
2. indextime latency
Is there a way to create standard timezones across all data coming into Splunk? Please assist.
This is a little confusing. What you are showing in the first screenshot is not a raw event, that is the output of some scheduled search or a notable event? How exactly is that relevant for this discussion?
Can you perhaps show the search you ran to get the table from the 2nd picture?
screenshot 1 - is the output of some scheduled search which is categorised as sourcetype stash, and it has latency in _indextime.
Following is the search I used, which triggered output from screenshot2
I would start by checking the timezone config of the originating hosts. Especially if the offset is clearly an (almost) exact nr. of hours.
Quite possibly a US host is sending data with timestamps in its local timezone (US Central, GMT-5) to your HF / Indexer(s) which you configured to interpret the data as GMT, causing this skew.
Ideally source systems are configured to report timezone in the timestamp, to prevent any confusion (and avoid specific config to adjust for offsets). If that is not an option, another solution can be to install a Heavy/Universal Forwarder in the same timezone as the initial collection point. If that is also not an option, you'll need to put specific config in place to assign the correct timezone (e.g. using host based stanzas in props.conf).
Source data is coming from timezone EST, hence I have manually added TZ=EST (also tried GMT-5) definiton for sepcific host on HF. Both are failed 😞
I am still seeing latency in indextime. As I mentioned earlier this is syslog data. Is there any other way I can fix this issue?
No, you cannot combine host and sourcetype criteria like that. It is either one or the other.
As for using 'EST', not sure if that is ideal. It is listed as deprecated:
This wiki page is referred in Splunk docs as the reference for acceptable time zone names, see: https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Applytimezoneoffsetstotimestamps
Also: if your source system is using DST during summer, better use for example
Also: make sure to deploy this config on the first HF that touches the data. And make sure the host field is actually populated with the value used in props.conf. Note that for syslog data splunk TAs often override the host value based on event content, so what you see in splunk web may not be the host name with which the data arrives at the HF originally.
Have you determined if that is a time extraction/time zone issue or a true lag of indexing?
Where the data that has this lag is coming from? Is that windows/linux logs or syslog? Or something else?
Usually if it is lag caused by some data collection issue, it will fluctuate. Any offsets that are pretty static and also very close to an exact nr of hours offset is likely a timezone config issue.
One way to be sure is to check on the affected host when the event really happened (assuming the host also stores the events locally for a while). Or trigger some specific event on the host and see when that becomes visible in Splunk and with what timestamp.
I can see data is coming from syslog. Below are few of the sourcetypes which also has index lagging
I also noticed data_zone associated with these hosts are either "local" or "0".