Getting Data In
Highlighted

_indextime is 5 hrs ahead of event time (_time)

New Member

Hi,

We have Splunk Enterprise 7.2.6 in our environment. I noticed there are latencies (difference between _time and _indextime from 1hr to 10hrs). My Splunk Heavy Forwarders are in GMT timezone, hence I have set TZ = UTC for few of the sourcetypes in props.conf of HF and it worked.

Still I am seeing time difference of 5hrs to 10hrs on few hosts for specific sourcetypes. I am unsure which is creating latency in _indextime. Attached screenshot for reference.

Can someone please assist me to fix this issue?

][1]

0 Karma
Highlighted

Re: _indextime is 5 hrs ahead of event time (_time)

Builder

Have you determined if that is a time extraction/time zone issue or a true lag of indexing?
Where the data that has this lag is coming from? Is that windows/linux logs or syslog? Or something else?

Highlighted

Re: _indextime is 5 hrs ahead of event time (_time)

New Member

I can see data is coming from syslog. Below are few of the sourcetypes which also has index lagging

  1. syslog
  2. stash

I also noticed data_zone associated with these hosts are either "local" or "0".

0 Karma
Highlighted

Re: _indextime is 5 hrs ahead of event time (_time)

New Member

Hi,

How can I determine if this is true lag of indexing? appreacite any suggestions.

0 Karma
Highlighted

Re: _indextime is 5 hrs ahead of event time (_time)

Ultra Champion

Usually if it is lag caused by some data collection issue, it will fluctuate. Any offsets that are pretty static and also very close to an exact nr of hours offset is likely a timezone config issue.

One way to be sure is to check on the affected host when the event really happened (assuming the host also stores the events locally for a while). Or trigger some specific event on the host and see when that becomes visible in Splunk and with what timestamp.

0 Karma
Highlighted

Re: _indextime is 5 hrs ahead of event time (_time)

Ultra Champion

I would start by checking the timezone config of the originating hosts. Especially if the offset is clearly an (almost) exact nr. of hours.

Quite possibly a US host is sending data with timestamps in its local timezone (US Central, GMT-5) to your HF / Indexer(s) which you configured to interpret the data as GMT, causing this skew.

Ideally source systems are configured to report timezone in the timestamp, to prevent any confusion (and avoid specific config to adjust for offsets). If that is not an option, another solution can be to install a Heavy/Universal Forwarder in the same timezone as the initial collection point. If that is also not an option, you'll need to put specific config in place to assign the correct timezone (e.g. using host based stanzas in props.conf).

Highlighted

Re: _indextime is 5 hrs ahead of event time (_time)

New Member

Hi,

Source data is coming from timezone EST, hence I have manually added TZ=EST (also tried GMT-5) definiton for sepcific host on HF. Both are failed 😞

I am still seeing latency in indextime. As I mentioned earlier this is syslog data. Is there any other way I can fix this issue?

0 Karma
Highlighted

Re: _indextime is 5 hrs ahead of event time (_time)

New Member

Also while defining TZ value, can I define as follows

[host::.... AND sourcetype::syslog]
TZ = EST

is this right defintion?

0 Karma
Highlighted

Re: _indextime is 5 hrs ahead of event time (_time)

Ultra Champion

No, you cannot combine host and sourcetype criteria like that. It is either one or the other.

As for using 'EST', not sure if that is ideal. It is listed as deprecated:
https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
This wiki page is referred in Splunk docs as the reference for acceptable time zone names, see: https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Applytimezoneoffsetstotimestamps

Also: if your source system is using DST during summer, better use for example America/New_York.

Also: make sure to deploy this config on the first HF that touches the data. And make sure the host field is actually populated with the value used in props.conf. Note that for syslog data splunk TAs often override the host value based on event content, so what you see in splunk web may not be the host name with which the data arrives at the HF originally.

0 Karma
Highlighted

Re: _indextime is 5 hrs ahead of event time (_time)

New Member

I have uploaded few more snapshots to this query. I am unsure if you have ever faced this issue. Could you please review and provide your suggestions on what can be done?

0 Karma