Getting Data In

Does Splunk support indexing of timestamps in tai64nlocal format?

Splunk Employee
Splunk Employee

I did not find this in the Splunk docs.

As a test, I just indexed some sample events with tai64nlocal format and it seems like works, for the most part.

After splunking the sample events:

7/9/10  8:53:29.000 AM  |  @400000004c3729d92f494b14 - msg
7/9/10  8:53:29.000 AM  |  @400000004c3729d92f516934 - msg
7/9/10  8:53:29.000 AM  |  @400000004c3729d92f5a46bc - msg
7/9/10 11:39:10.000 AM  |  @400000004c3750ae28a07d64 - msg
7/9/10 11:46:39.000 AM  |  @400000004c37526f24b9b1fc - msg
7/9/10 11:46:39.000 AM  |  @400000004c37526f24ba7164 - msg
7/9/10 11:46:39.000 AM  |  @400000004c37526f24ba7d1c - msg
7/9/10 11:47:42.000 AM  |  @400000004c3752ae16f42ebc - msg
7/9/10 11:47:42.000 AM  |  @400000004c3752ae16f4e654 - msg
7/9/10 11:47:42.000 AM  |  @400000004c3752ae16f4f5f4 - msg

After running sample events through daemontools:

2010-07-09 09:53:19.793332500 - msg
2010-07-09 09:53:19.793864500 - msg
2010-07-09 09:53:19.794445500 - msg
2010-07-09 12:39:00.681606500 - msg
2010-07-09 12:46:29.616149500 - msg
2010-07-09 12:46:29.616198500 - msg
2010-07-09 12:46:29.616201500 - msg
2010-07-09 12:47:32.385101500 - msg
2010-07-09 12:47:32.385148500 - msg
2010-07-09 12:47:32.385152500 - msg

BTW, my sample events are from a server in a timezone that is one hour ahead of mine, so the timestamps inside Splunk are an hour off, as you can see above. Also notice that the minutes are ten seconds off, as if Splunk is rounding up, commpared to what daemontools shows.

Overall, pretty close, though.

Wondering if anyone can confirm we support it and explain the ten seconds offset?

Communicator

Hi,

Just having the same issue here with Splunk 4.2.1 but Splunk still ignores the milliseconds.

The datetime.xml mentions a "subsecond" for the utcepoch, but I don't know how to use it.

Splunk seems to recognise only the first 16 characters as mentioned previously. I then tried to remove the "16" in the regex in the datetime.xml ( ^@[da-fA-F]{16,24} ), but this didn't help neither.

Any idea anyone?

Regards,
Olivier

0 Karma

Splunk Employee
Splunk Employee

As you say, for the most part, Splunk is reading it correctly, although Splunk doesn't seem to be considering the subseconds. It appears that Splunk does accept tai64, though I note that only this part of the timestamp (@400000004c3729d9) is read, the rest of it represents subseconds.

Maybe a custom datetime.xml would let it get the subseconds too? Worth a try, at least.

If you do the actual math the timestamp @400000004c3729d9 is hex 0x4c3729d9 seconds past 1970/01/01:00:00:00, which is 1278683609 in decimal, which (taken modulo 60) is 29 seconds past the minute.

The ten second difference has something to do with leap seconds:

http://en.wikipedia.org/wiki/Leap_second

Therefore, all things considered, Splunk is correct if we take it as TAI format used to represent UTC (which is what unix epoch is in practice), vs if it's TAI format supposed to represent actual TAI, in which case, both Splunk and daemontools are actually wrong because neither is correcting for leap seconds (but from different start dates).

-- posted on behalf of Gerald K, King of Splunk Answers --

State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!