Why is _time indicating UTC when the event happened in EST.

[djreschke]

_time: 2021-11-19T11:34:02.000+0000

date_hour: 11

date_mday: 19

date_wday: friday  

date_year: 2021

date_zone: -300

raw log snippet

[19/Nov/2021:11:34:02 -0500]

2021-11-19T11:34:02.000+0000 indicates UTC.

Does this indicate timezone? 

[PickleRick]

There are various things that are happening here.

Firstly, the date_* fields - quoting from the docs (https://docs.splunk.com/Documentation/Splunk/latest/Data/Aboutdefaultfields😞

"Only events that have timestamp information in them as generated by their respective systems will have date_* fields. If an event has a date_* field, it represents the value of time/date directly from the event itself. If you have specified any timezone conversions or changed the value of the time/date at indexing or input time (for example, by setting the timestamp to be the time at index or input time), these fields will not represent that."

So don't rely too much on the date_* fields because they don't have to match (as you can see) the final parsed _time value.

Secondly, if you don't have your date format specified, splunk tries to find the date on its own but doesn't always do it properly (especially if the date is expressed somewhat exoticaly). Also, finding timestamp is one of the "heaviest" part of ingestion pipeline. So it's best to specify explicitly where the timestamp is within the event by means of TIME_PREFIX in props.conf and what is the timestamp format with TIME_FORMAT. If there is no timezone within the event itself you can also set TZ for a given sourcetype.

[djreschke]

@PickleRick 

All of the time information is being parsed correctly, it is simply not so simply different on the _time field from the Web UI to the exported csv. 

Web UI _time is showing: 2021-11-19T11:34:02.000-05:00

CSV Export _time is showing: 2021-11-19T11:34:02.000+0000

I never noticed the difference before today but for users that don't log into Splunk, they are assuming that the CSV _time is in UTC, so they are converting this to EST which puts this event at 630 in the morning, which is not correct. 

[PickleRick]

That's interesting, because indeed splunk does make something strange with _time export on CSV... when it's formated by default.

If I did simply

| makeresults

I'd get my results as

When I exported the job to csv, I'd get

"_time"
"2021-11-19T20:49:41.000+0200"

But if I started fooling around with fieldformat I started getting weird results:

| makeresults 
| eval t=_time
| fieldformat t=strftime(t,"%Y-%m-%d %H:%M:%S.%l %z")

As you can see, both fields, t and _time should have the same value. And in the WebUI it does indeed seem so:

But the CSV export shows...

"_time",t
"2021-11-19T20:52:30.000+0200","2021-11-19 20:52:30.000 +0100"

So if we render the _time without the timezone information, CSV export produces the default timezone on its own anyway.

But if we render the _time with a proper timezone including format...

| makeresults 
| eval t=_time
| fieldformat t=strftime(t,"%Y-%m-%d %H:%M:%S.%l %z")
| fieldformat _time=strftime(_time,"%Y-%m-%d %H:%M:%S.%l %z")

Again - t and _time are equal but this time also string representations are explicitly created with the same format, including timezone information.

And now the CSV includes proper TZ information in both fields.

"_time",t
"2021-11-19 20:55:40.000 +0100","2021-11-19 20:55:40.000 +0100"

So it seems it's not the webUI that is at fault but there's something "wrong" with CSV export.

[djreschke]

@PickleRick 

Agreed, the UI is correct, I can't post screenshots. I have open a support ticket with Splunk. Thank you for validating that there is a difference. I'll keep this post updated as I here back from Support. 

[djreschke]

@PickleRick 

What version of Splunk are you running? I am running 8.1.1. 

[PickleRick]

Version:8.2.2.1

Build:ae6821b7c64b

[vhharanpositka]

Hi @djreschke 

This is based on the default timezone of the Splunk server.

The time zone can be changed based on the steps in the link below.

Set your time zone

Choose the time zone in which you view events, anomalies, and threats.

1. Select your username from the menu.
2. Click Profile.
3. Select Preferences.
4. Select a Time Zone of UTC or Local. The local time zone is detected based on your web browser settings.
5. Click OK to save.

 Reference Link: https://docs.splunk.com/Documentation/UBA/5.0.5/User/Profile

Regards

[djreschke]

My timezone is in EST, so I would imagine the it would show something like -500?

[vhharanpositka]

Yes, it is right

[djreschke]

Can you please clarify some more? 

Its right that is showing 2021-11-19T11:34:02.000+0000

Or is this wrong from based of my timezone. 

[djreschke]

So when I export the log the it what it shows as _time, but i am seeing the correct _time in the field on the Search head. 

Is this a know thing that it drops the timezone when you export a log.