Difference between _time and time shown by Splunk interface

[SplunkExplorer]

Hi Splunkers, I have a problem with timestamp on our platform. Here some assumption and acquired knowledge.

Knowledge

• _time =  is the event time (the time which is present in the event. In other words: the time when the event was generated.
• _indextime = is the index time or, if you prefer, the time when the events have been indexed.
• Issue with timezone shown can be related to user settings, that can be changed under username -> Preferences -> Timezone.

Environment: a Splunk Cloud SaaS platform with logs ingested in different ways:

• Forwarder (both UF and HF)
• API
• Syslog
• File monitoring

Issue:

If I expand the event and I examinate the _time field: 

Why, in my case, time event and time shown are different?

Important additional Info

• Our user settings timezone are set on GMT+1 (due we are in Italy) for all users.
• You see a Windows events as sample, but the problem is present on all logs: it doesn't matter what log source I consider and how it is sending events to Splunk. Every log show time difference.
• The difference between _time and time shown is always on 1 hour, for every events on every log sources.

I searched here on community and I found other topics about this issue, some of them has been very useful to gain a basic knowledge like Difference Between Event Time and _time  but, due we are on cloud (with limited chance to set some file and parameter that are involved) and the issue is for all events, I'm still locked on this problem. 

[PickleRick]

Quoting the docs:

How Splunk software determines time zones

To determine the time zone to assign to a timestamp, Splunk software uses the following logic in order of precedence:

1. Use the time zone specified in raw event data (for example, PST, -0800), if present.
2. Use the TZ attribute set in props.conf, if the event matches the host, source, or source type that the stanza specifies.
3. If the forwarder and the receiving indexer are version 6.0 or higher, use the time zone that the forwarder provides.
4. Use the time zone of the host that indexes the event.

From my experience with Windows (I see windows events format) the most common error is when someone forgets to set up a system timezone on install and as a result - the whole server is indeed in a wrong timezone and effectively uses wrong time. Otherwise windows events are properly ingested and parsed (I assume you have TA_windows on your receiving indexers or HF).

[SplunkExplorer]

Thanks a lot @PickleRick. 

Regarding Windows, we have UF installed on each Data sources; they sent file to a dedicated HF that then forward data to Splunk Cloud.

[PickleRick · Accepted solution]

It's typically enough to have

1) Well configured timezone on the server itself

2) You must have the TA_windows on the HF for proper index-time parsing (of course you also need it on SHs - in your case - in your Cloud instance for search-time extractions, eventtypes and so on but that's another story).

[ITWhisperer]

_indextime is not being represented in your screen shot.

It looks like your event, which contains the text "01/24/2023 09:42:07 AM" (without any timezone information) is being interpreted as UTC i.e. GMT+0. This is converted the UTC epoch time (number of seconds) and stored in _time.

_time is then displayed in the event view in local time i.e. GMT+1 so 09:... becomes 08:... hence your "hour difference".

[SplunkExplorer]

Hi @ITWhisperer, thanks for your answer. Your answer give me the root cause and that is fine; now the question is: how should I fix this?

[richgalloway]

Review how the data is ingested.  By default, Splunk Cloud presumes all event times are UTC.  That means all non-UTC timestamps must be identified as such.  The TIME_FORMAT setting in props.conf should include the time zone if the event timestamp does (your sample event does not).  Other events should use the TZ setting in props.conf to specify the time zone.

Every sourcetype onboarded should have props.conf settings to avoid having Splunk make incorrect assumptions about the data.

[SplunkExplorer]

Very useful @richgalloway. Thanks a lot.