Getting Data In

Time zone in TIME_FORMAT

vbumgarner
Contributor

Given this timestamp:

01/Mar/2011:17:25:49.666+0000

What is the right format?
I'm leaning towards:

TIME_FORMAT = %d/%b/%Y:$H:%M:%S.%3N+%z

but I'm not quite sure what this means from the extended strptime documentation:

%z, %::z, %:::z GNU libc support.
Tags (1)
0 Karma
1 Solution

Stephen_Sorkin
Splunk Employee
Splunk Employee

Unfortunately we don't ship our "DateTimeTest" utility that allows easy testing of strptime/strftime functionality, please file an ER for that to be included from the CLI/UI. Here's my findings (first note that you have $H rather than %H):

[ssorkin@MrT current]$ src/util/tests/DateTimeTest '%d/%b/%Y:%H:%M:%S.%3N+%z' '01/Mar/2011:17:25:49.666+0000'
01/Mar/2011:09:25:49.666+-0800
[ssorkin@MrT current]$ src/util/tests/DateTimeTest '%d/%b/%Y:%H:%M:%S.%3N%z' '01/Mar/2011:17:25:49.666+0000'
01/Mar/2011:09:25:49.666-0800

So basically, %z does essentially the right thing, but the + is unnecessary and actually causes wrong results.

As a clarification, the : in %z says how to split the time zone offset. One : looks like -08:00 for PST, two : looks like -08:00:00.

View solution in original post

gjanders
SplunkTrust
SplunkTrust

After reading about %z on these pages and http://docs.splunk.com/Documentation/Splunk/6.4.3/Data/Configuretimestamprecognition , I found the python 3 documentation https://docs.python.org/3/library/datetime.html has this explanation:
"
%z UTC offset in the form +HHMM or -HHMM (empty string if the object is naive). (empty), +0000, -0400, +1030 (6)
%Z Time zone name (empty string if the object is naive). (empty), UTC, EST, CST

"

The documentation has been prompt updated to reflect this information correctly!
Furthermore, the preview functionality mentioned is now part of the "add data"/data upload option in Splunk...

peterzhoupeterz
Engager

%z is -0400 This format is not standard. if your machine is configure as Eastern Date Time

%Z is EDT if your machine is configure as Eastern Date Time, not too much use for storing it in data base. By the way I live in New York.

%:z is -04:00 That is the one most useful in hours and minutes. It can be used across computer languages

%::z is -04:00:00 It is over kill. we don't need second for time zone

%:::z is even more over kill, no use in reality

emma
Splunk Employee
Splunk Employee

Actually, you should use the new preview feature (shipped in 4.3) to test any tz configs.

0 Karma

bwooden
Splunk Employee
Splunk Employee

Actually, that's true. Though actually, it was posted 2 years later.

Preview is great for manipulation & validation of timestamp extraction before implementation.

Also: http://docs.splunk.com/Documentation/Splunk/latest/Data/Configuretimestamprecognition#Enhanced_strpt...

0 Karma

Stephen_Sorkin
Splunk Employee
Splunk Employee

Unfortunately we don't ship our "DateTimeTest" utility that allows easy testing of strptime/strftime functionality, please file an ER for that to be included from the CLI/UI. Here's my findings (first note that you have $H rather than %H):

[ssorkin@MrT current]$ src/util/tests/DateTimeTest '%d/%b/%Y:%H:%M:%S.%3N+%z' '01/Mar/2011:17:25:49.666+0000'
01/Mar/2011:09:25:49.666+-0800
[ssorkin@MrT current]$ src/util/tests/DateTimeTest '%d/%b/%Y:%H:%M:%S.%3N%z' '01/Mar/2011:17:25:49.666+0000'
01/Mar/2011:09:25:49.666-0800

So basically, %z does essentially the right thing, but the + is unnecessary and actually causes wrong results.

As a clarification, the : in %z says how to split the time zone offset. One : looks like -08:00 for PST, two : looks like -08:00:00.

woodcock
Esteemed Legend

I made a request to ship your tool here:
https://ideas.splunk.com/ideas/EID-I-59

0 Karma

to4kawa
Ultra Champion
| makeresults 
| eval time1="01/Mar/2011:17:25:49.666+0000" 
| appendpipe 
    [ eval time2="2015-10-22T22:41:52.546249+00:00"] 
| eval _time=coalesce(strptime(time2,"%FT%T.%6Q%:z"),strptime(time1,"%d/%B/%Y:%T.%3Q%z"))

on Splunk ver8.0.1

btool [options] {check|validate-strptime|validate-regex}
Is this not enough?

0 Karma

blahblah
New Member

Rather than submitting yet another question about timestamp formating, could you please update your answer with the correct format string to successfully parse a timestamp like 2015-10-22T22:41:52.546249+00:00? I tried %Y-%m-%dT%H:%M:%S.%6N%:z, but it doesn't seems to be working. The timezone info is probably not being captured as all event times are being translated as if the event timezone (always UTC) was the same as the splunk server (-0500), which, translated to my splunk user timezone (-0200), gives me 2015-10-23T01:41:52.546249-02:00. Any clue?

0 Karma

vbumgarner
Contributor

Perfect. Thanks.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...