topic Re: Timestamp recognition in Getting Data In

Timestamp recognition

elaine0102 — Mon, 28 Sep 2020 12:33:26 GMT

Hi, I have successfuly extracted a timestamp. However, I need Splunk to recognised it as the event timestamp.
Please kindly guide me what should I do to achieve that.
Thank you.

e.g. created_time = 2012-09-26T02:59:57+0000

TIME_PREFIX = created_time
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 27

Re: Timestamp recognition

Ayn — Thu, 04 Oct 2012 05:25:33 GMT

You don't really need to use the TIME_PREFIX unless there are other timestamps before this one in your event, or if this timestamp is located very far into the event text. Similarly you don't need to change MAX_TIMESTAMP_LOOKAHEAD from its default unless you have a good reason to do so.

Other than that, your TIME_FORMAT looks fine. What are your current results with these settings?

UPDATE: So, I did a second check - it seems Python's strptime (which is what Splunk uses) is pretty picky about that you need to specify how it should handle ALL of the string you want to parse. In your case, that string would be "2012-09-26T02:59:57+0000", and Python's strptime complains about the trailing "+0000":

>>> time.strptime("2012-09-26T02:59:57+0000","%Y-%m-%dT%H:%M:%S")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.6/_strptime.py", line 454, in _strptime_time
    return _strptime(data_string, format)[0]
  File "/usr/local/lib/python2.6/_strptime.py", line 328, in _strptime
    data_string[found.end():])
ValueError: unconverted data remains: +0000

This trailing +0000 can be referred to in some strptime implementations as %z - however, unfortunately Python does not support this identifier. The workaround I can think of is to specify a combination of TIME_PREFIX and MAX_TIMESTAMP_LOOKAHEAD that make sure that ONLY the part of the timestamp before the +0000 is included (i.e. "2012-09-26T02:59:57").

Re: Timestamp recognition

elaine0102 — Mon, 28 Sep 2020 12:33:33 GMT

Thanks for replying.
Yes, I do need TIME_PREFIX and you are right, I do not need to change the default MAX_TIMESTAMP_LOOKAHEAD.

My current result is that Splunk has successfully extract the timestamp that I want. However, on Splunk search, it does not show the timestamp that I extracted as my event time.

What I should go about to make that as my event time?

Re: Timestamp recognition

Ayn — Thu, 04 Oct 2012 06:01:45 GMT

If Splunk extracted your timestamp correctly, what do you mean by that it is not your event time? Do you mean that you have created a separate extraction that extracts this timestamp as a FIELD, but on top of that you want Splunk to use this as the event timestamp as well?

Which timestamp does Splunk currently use for these events? If you check your splunkd.log, do you see any errors related to the timestamp extraction for this source?

Re: Timestamp recognition

elaine0102 — Mon, 28 Sep 2020 12:33:42 GMT

Sorry, just realised that I encountered this warning when I preview my data with the following setting:

TIME_PREFIX = created_time
TIME_FORMAT = %Y-%m-%dT%H:%M:%S

Warning: Could not use strptime to parse timestamp from
" = 2012-09-26T02:59:57+0000 , updated_time = 2012-10-01T01:40:34+0000 counter:0".

Re: Timestamp recognition

elaine0102 — Mon, 28 Sep 2020 12:33:47 GMT

Pardon me for my confusing issue. Created_time is the one I want it to be my event_time.

Previously, I have overlooked the warning of timestamp and had the created_time became one of the interesting field.

Thus, I guessed that is why my event_time is the time when I added the data into Splunk and not the created_time as my event_time.

Re: Timestamp recognition

Ayn — Mon, 28 Sep 2020 12:34:11 GMT

I checked your time format against the sample date you provided, and it works just like it should. What's your current config? Did you remove the TIME_PREFIX and MAX_TIMESTAMP_LOOKAHEAD?

Re: Timestamp recognition

elaine0102 — Mon, 28 Sep 2020 12:34:14 GMT

current config:
TIME_PREFIX = created_time
TIME_FORMAT = %Y-%m-%dT%H:%M:%S

I did not remove TIME_PREFIX as I do need it.
As the timestamp is located very far into the event text.

Re: Timestamp recognition

Ayn — Fri, 05 Oct 2012 09:43:10 GMT

OK. I did some more tests - see my updated answer.

Re: Timestamp recognition

elaine0102 — Mon, 28 Sep 2020 12:34:57 GMT

I tried the following config, however it does not work too:

TIME_PREFIX = created_time
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 23

Re: Timestamp recognition

Ayn — Mon, 08 Oct 2012 04:50:01 GMT

I believe you would include the + character with that MAX_TIMESTAMP_LOOKAHEAD (unless you have a space after "created_time"). Check splunkd.log to see what it says about its strptime parsing attempts.

Re: Timestamp recognition

BobM — Wed, 22 May 2013 16:25:39 GMT

Assuming the number after the + is 10,000ths of a second you can tell splunk to use this with :-

TIME_FORMAT = %Y-%m-%dT%H:%M:%S+%4N

If it is consistently +0000 you can specify it in the time format.

TIME_FORMAT = %Y-%m-%dT%H:%M:%S+0000