topic Why is _time indicating UTC when the event happened in EST. in Splunk Search

Why is _time indicating UTC when the event happened in EST.

djreschke — Fri, 19 Nov 2021 17:27:10 GMT

_time: 2021-11-19T11:34:02.000+0000

date_hour: 11

date_mday: 19

date_wday: friday

date_year: 2021

date_zone: -300

raw log snippet

[19/Nov/2021:11:34:02 -0500]

2021-11-19T11:34:02.000+0000 indicates UTC.

Does this indicate timezone?

Re: Why is _time indicating UTC when the event happened in EST.

vhharanpositka — Fri, 19 Nov 2021 17:35:56 GMT

Hi @djreschke

This is based on the default timezone of the Splunk server.

The time zone can be changed based on the steps in the link below.

Set your time zone

Choose the time zone in which you view events, anomalies, and threats.

Select your username from the menu.
Click Profile.
Select Preferences.
Select a Time Zone of UTC or Local. The local time zone is detected based on your web browser settings.
Click OK to save.

Reference Link: https://docs.splunk.com/Documentation/UBA/5.0.5/User/Profile

Regards

Re: Why is _time indicating UTC when the event happened in EST.

djreschke — Fri, 19 Nov 2021 17:39:49 GMT

My timezone is in EST, so I would imagine the it would show something like -500?

Re: Why is _time indicating UTC when the event happened in EST.

vhharanpositka — Fri, 19 Nov 2021 17:41:20 GMT

Yes, it is right

Re: Why is _time indicating UTC when the event happened in EST.

djreschke — Fri, 19 Nov 2021 17:52:00 GMT

Can you please clarify some more?

Its right that is showing 2021-11-19T11:34:02.000+0000

Or is this wrong from based of my timezone.

Re: Why is _time indicating UTC when the event happened in EST.

djreschke — Fri, 19 Nov 2021 18:26:16 GMT

So when I export the log the it what it shows as _time, but i am seeing the correct _time in the field on the Search head.

Is this a know thing that it drops the timezone when you export a log.

Re: Why is _time indicating UTC when the event happened in EST.

PickleRick — Fri, 19 Nov 2021 18:38:22 GMT

There are various things that are happening here.

Firstly, the date_* fields - quoting from the docs (https://docs.splunk.com/Documentation/Splunk/latest/Data/Aboutdefaultfields😞

"Only events that have timestamp information in them as generated by their respective systems will have date_* fields. If an event has a date_* field, it represents the value of time/date directly from the event itself. If you have specified any timezone conversions or changed the value of the time/date at indexing or input time (for example, by setting the timestamp to be the time at index or input time), these fields will not represent that."

So don't rely too much on the date_* fields because they don't have to match (as you can see) the final parsed _time value.

Secondly, if you don't have your date format specified, splunk tries to find the date on its own but doesn't always do it properly (especially if the date is expressed somewhat exoticaly). Also, finding timestamp is one of the "heaviest" part of ingestion pipeline. So it's best to specify explicitly where the timestamp is within the event by means of TIME_PREFIX in props.conf and what is the timestamp format with TIME_FORMAT. If there is no timezone within the event itself you can also set TZ for a given sourcetype.

Re: Why is _time indicating UTC when the event happened in EST.

djreschke — Fri, 19 Nov 2021 18:47:46 GMT

@PickleRick

All of the time information is being parsed correctly, it is simply not so simply different on the _time field from the Web UI to the exported csv.

Web UI _time is showing: 2021-11-19T11:34:02.000-05:00

CSV Export _time is showing: 2021-11-19T11:34:02.000+0000

I never noticed the difference before today but for users that don't log into Splunk, they are assuming that the CSV _time is in UTC, so they are converting this to EST which puts this event at 630 in the morning, which is not correct.

Re: Why is _time indicating UTC when the event happened in EST.

PickleRick — Fri, 19 Nov 2021 19:58:52 GMT

That's interesting, because indeed splunk does make something strange with _time export on CSV... when it's formated by default.

If I did simply

| makeresults

I'd get my results as

When I exported the job to csv, I'd get

"_time"
"2021-11-19T20:49:41.000+0200"

But if I started fooling around with fieldformat I started getting weird results:

| makeresults 
| eval t=_time
| fieldformat t=strftime(t,"%Y-%m-%d %H:%M:%S.%l %z")

As you can see, both fields, t and _time should have the same value. And in the WebUI it does indeed seem so:

But the CSV export shows...

"_time",t
"2021-11-19T20:52:30.000+0200","2021-11-19 20:52:30.000 +0100"

So if we render the _time without the timezone information, CSV export produces the default timezone on its own anyway.

But if we render the _time with a proper timezone including format...

| makeresults 
| eval t=_time
| fieldformat t=strftime(t,"%Y-%m-%d %H:%M:%S.%l %z")
| fieldformat _time=strftime(_time,"%Y-%m-%d %H:%M:%S.%l %z")

Again - t and _time are equal but this time also string representations are explicitly created with the same format, including timezone information.

And now the CSV includes proper TZ information in both fields.

"_time",t
"2021-11-19 20:55:40.000 +0100","2021-11-19 20:55:40.000 +0100"

So it seems it's not the webUI that is at fault but there's something "wrong" with CSV export.

Re: Why is _time indicating UTC when the event happened in EST.

djreschke — Fri, 19 Nov 2021 20:04:37 GMT

@PickleRick

Agreed, the UI is correct, I can't post screenshots. I have open a support ticket with Splunk. Thank you for validating that there is a difference. I'll keep this post updated as I here back from Support.

Re: Why is _time indicating UTC when the event happened in EST.

djreschke — Mon, 22 Nov 2021 13:30:31 GMT

@PickleRick

What version of Splunk are you running? I am running 8.1.1.

Re: Why is _time indicating UTC when the event happened in EST.

PickleRick — Mon, 22 Nov 2021 19:06:08 GMT

Version:8.2.2.1

Build:ae6821b7c64b