Splunk Search

Why is _time indicating UTC when the event happened in EST.

djreschke
Communicator

_time: 2021-11-19T11:34:02.000+0000

date_hour: 11

date_mday: 19

date_wday: friday  

date_year: 2021

date_zone: -300

raw log snippet

[19/Nov/2021:11:34:02 -0500]

2021-11-19T11:34:02.000+0000 indicates UTC.

Does this indicate timezone? 

Labels (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

There are various things that are happening here.

Firstly, the date_* fields - quoting from the docs (https://docs.splunk.com/Documentation/Splunk/latest/Data/Aboutdefaultfields😞

"Only events that have timestamp information in them as generated by their respective systems will have date_* fields. If an event has a date_* field, it represents the value of time/date directly from the event itself. If you have specified any timezone conversions or changed the value of the time/date at indexing or input time (for example, by setting the timestamp to be the time at index or input time), these fields will not represent that."

So don't rely too much on the date_* fields because they don't have to match (as you can see) the final parsed _time value.

Secondly, if you don't have your date format specified, splunk tries to find the date on its own but doesn't always do it properly (especially if the date is expressed somewhat exoticaly). Also, finding timestamp is one of the "heaviest" part of ingestion pipeline. So it's best to specify explicitly where the timestamp is within the event by means of TIME_PREFIX in props.conf and what is the timestamp format with TIME_FORMAT. If there is no timezone within the event itself you can also set TZ for a given sourcetype.

djreschke
Communicator

@PickleRick 

All of the time information is being parsed correctly, it is simply not so simply different on the _time field from the Web UI to the exported csv. 

Web UI _time is showing: 2021-11-19T11:34:02.000-05:00

CSV Export _time is showing: 2021-11-19T11:34:02.000+0000

I never noticed the difference before today but for users that don't log into Splunk, they are assuming that the CSV _time is in UTC, so they are converting this to EST which puts this event at 630 in the morning, which is not correct. 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

That's interesting, because indeed splunk does make something strange with _time export on CSV... when it's formated by default.

If I did simply

| makeresults

I'd get my results as

PickleRick_0-1637351403712.png

When I exported the job to csv, I'd get

"_time"
"2021-11-19T20:49:41.000+0200"

But if I started fooling around with fieldformat I started getting weird results:

| makeresults 
| eval t=_time
| fieldformat t=strftime(t,"%Y-%m-%d %H:%M:%S.%l %z")

As you can see, both fields, t and _time should have the same value. And in the WebUI it does indeed seem so:

PickleRick_1-1637351630205.png

But the CSV export shows...

"_time",t
"2021-11-19T20:52:30.000+0200","2021-11-19 20:52:30.000 +0100"

So if we render the _time without the timezone information, CSV export produces the default timezone on its own anyway.

But if we render the _time with a proper timezone including format...

| makeresults 
| eval t=_time
| fieldformat t=strftime(t,"%Y-%m-%d %H:%M:%S.%l %z")
| fieldformat _time=strftime(_time,"%Y-%m-%d %H:%M:%S.%l %z")

Again - t and _time are equal but this time also string representations are explicitly created with the same format, including timezone information.

PickleRick_2-1637351840075.png

 

And now the CSV includes proper TZ information in both fields.

"_time",t
"2021-11-19 20:55:40.000 +0100","2021-11-19 20:55:40.000 +0100"

So it seems it's not the webUI that is at fault but there's something "wrong" with CSV export.

djreschke
Communicator

@PickleRick 

Agreed, the UI is correct, I can't post screenshots. I have open a support ticket with Splunk. Thank you for validating that there is a difference. I'll keep this post updated as I here back from Support. 

djreschke
Communicator

@PickleRick 

 

What version of Splunk are you running? I am running 8.1.1. 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Version:8.2.2.1

Build:ae6821b7c64b

vhharanpositka
Path Finder

Hi @djreschke 

 

This is based on the default timezone of the Splunk server.

The time zone can be changed based on the steps in the link below.

 

Set your time zone

Choose the time zone in which you view events, anomalies, and threats.

  1. Select your username from the menu.
  2. Click Profile.
  3. Select Preferences.
  4. Select a Time Zone of UTC or Local. The local time zone is detected based on your web browser settings.
  5. Click OK to save.

 Reference Link: https://docs.splunk.com/Documentation/UBA/5.0.5/User/Profile

 

Regards

0 Karma

djreschke
Communicator

My timezone is in EST, so I would imagine the it would show something like -500?

 

 

0 Karma

vhharanpositka
Path Finder

Yes, it is right

0 Karma

djreschke
Communicator

Can you please clarify some more? 

 

Its right that is showing 2021-11-19T11:34:02.000+0000

Or is this wrong from based of my timezone. 

0 Karma

djreschke
Communicator

So when I export the log the it what it shows as _time, but i am seeing the correct _time in the field on the Search head. 

Is this a know thing that it drops the timezone when you export a log. 

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...