topic Re: Difference between _time and time shown by Splunk interface in Splunk Cloud Platform

Difference between _time and time shown by Splunk interface

SplunkExplorer — Fri, 26 Jan 2024 13:50:09 GMT

Hi Splunkers, I have a problem with timestamp on our platform. Here some assumption and acquired knowledge.

Knowledge

_time = is the event time (the time which is present in the event. In other words: the time when the event was generated.
_indextime = is the index time or, if you prefer, the time when the events have been indexed.
Issue with timezone shown can be related to user settings, that can be changed under username -> Preferences -> Timezone.

Environment: a Splunk Cloud SaaS platform with logs ingested in different ways:

Forwarder (both UF and HF)
API
Syslog
File monitoring

Issue:

If I expand the event and I examinate the _time field:

Why, in my case, time event and time shown are different?

Important additional Info

Our user settings timezone are set on GMT+1 (due we are in Italy) for all users.
You see a Windows events as sample, but the problem is present on all logs: it doesn't matter what log source I consider and how it is sending events to Splunk. Every log show time difference.
The difference between _time and time shown is always on 1 hour, for every events on every log sources.

I searched here on community and I found other topics about this issue, some of them has been very useful to gain a basic knowledge like Difference Between Event Time and _time but, due we are on cloud (with limited chance to set some file and parameter that are involved) and the issue is for all events, I'm still locked on this problem.

Re: Difference between _time and time shown by Splunk interface

ITWhisperer — Fri, 26 Jan 2024 14:01:26 GMT

_indextime is not being represented in your screen shot.

It looks like your event, which contains the text "01/24/2023 09:42:07 AM" (without any timezone information) is being interpreted as UTC i.e. GMT+0. This is converted the UTC epoch time (number of seconds) and stored in _time.

_time is then displayed in the event view in local time i.e. GMT+1 so 09:... becomes 08:... hence your "hour difference".

Re: Difference between _time and time shown by Splunk interface

SplunkExplorer — Fri, 26 Jan 2024 14:11:42 GMT

Hi @ITWhisperer, thanks for your answer. Your answer give me the root cause and that is fine; now the question is: how should I fix this?

Re: Difference between _time and time shown by Splunk interface

richgalloway — Fri, 26 Jan 2024 14:37:10 GMT

Review how the data is ingested. By default, Splunk Cloud presumes all event times are UTC. That means all non-UTC timestamps must be identified as such. The TIME_FORMAT setting in props.conf should include the time zone if the event timestamp does (your sample event does not). Other events should use the TZ setting in props.conf to specify the time zone.

Every sourcetype onboarded should have props.conf settings to avoid having Splunk make incorrect assumptions about the data.

Re: Difference between _time and time shown by Splunk interface

PickleRick — Fri, 26 Jan 2024 15:12:30 GMT

Quoting the docs:

How Splunk software determines time zones

To determine the time zone to assign to a timestamp, Splunk software uses the following logic in order of precedence:

Use the time zone specified in raw event data (for example, PST, -0800), if present.
Use the TZ attribute set in props.conf, if the event matches the host, source, or source type that the stanza specifies.
If the forwarder and the receiving indexer are version 6.0 or higher, use the time zone that the forwarder provides.
Use the time zone of the host that indexes the event.

From my experience with Windows (I see windows events format) the most common error is when someone forgets to set up a system timezone on install and as a result - the whole server is indeed in a wrong timezone and effectively uses wrong time. Otherwise windows events are properly ingested and parsed (I assume you have TA_windows on your receiving indexers or HF).

Re: Difference between _time and time shown by Splunk interface

SplunkExplorer — Fri, 26 Jan 2024 16:29:19 GMT

Thanks a lot @PickleRick.

Regarding Windows, we have UF installed on each Data sources; they sent file to a dedicated HF that then forward data to Splunk Cloud.

Re: Difference between _time and time shown by Splunk interface

SplunkExplorer — Fri, 26 Jan 2024 16:29:50 GMT

Very useful @richgalloway. Thanks a lot.

Re: Difference between _time and time shown by Splunk interface

PickleRick — Fri, 26 Jan 2024 17:40:07 GMT

It's typically enough to have

1) Well configured timezone on the server itself

2) You must have the TA_windows on the HF for proper index-time parsing (of course you also need it on SHs - in your case - in your Cloud instance for search-time extractions, eventtypes and so on but that's another story).