topic Re: Another timestamp question in Getting Data In

Another timestamp question

Greg_LeBlanc — Thu, 06 Oct 2011 19:07:45 GMT

I am having a difficult time extracting the correct timestamp from a specific log.

As you can see below, the beginning of the log entry there are two timestamps back to back.

2851,10/06/2011,18:59:29,10/06/2011,14:59:29,1011,

Here is my props.conf

    [sourcetype]
    MAX_TIMESTAMP_LOOKAHEAD = 20
    TZ = America/New_York
    TIME_PREFIX = \d{4},\d{2}/\d{2}/\d{4},\d{2}\:\d{2}\:\d{2},

The first timestamp is still being extracted. See anything wrong here?

EDIT: Can I perform these actions on a Universal Forwarder? Now that I think about it, I can't. Only a Heavy forwarder, correct?

EDIT: Looks like this was my fault. I had been trying this on a Universal Forwarder - I moved this to my Heavy Forwarder and it works! Thanks.

Re: Another timestamp question

jdunlea_splunk — Mon, 28 Sep 2020 09:57:18 GMT

As far as I can see your MAX_TIMESTAMP_LOOKAHEAD = 20 only gets you as far as "9:29,10/06/2011,14:59:29,1011,"

Try changing your MAX_TIMESTAMP_LOOKAHEAD=25.

Re: Another timestamp question

jbsplunk — Mon, 28 Sep 2020 09:57:21 GMT

Your TIME_PREFIX looks okay in my estimation. It also looks like your MAX_TIMESTAMP_LOOKAHEAD is correct. If that isn't working, I would suggest you try specifying TIME_FORMAT as well. This needs to be done where the data is parsed, so on the Heavy Forwarder, or on the indexer which the forwarder is sending, presuming a forwarder is part of the picture.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Configuretimestamprecognition

I think that this should be the correct format:

TIME_FORMAT = %m/%d/%Y,%H:%M:%S

Re: Another timestamp question

jdunlea_splunk — Mon, 28 Sep 2020 09:57:24 GMT

Oops i got confused on this! Indeed your MAX_TIMESTAMP_LOOKAHEAD is correct.

Re: Another timestamp question

jbsplunk — Mon, 28 Sep 2020 09:57:27 GMT

This is actually not correct. Using TIME_PREFIX tells splunk what precedes the timestamp. When MAX_TIMESTAMP_LOOKAHEAD is used in conjunction with TIME_PREFIX, you are telling splunk to look ahead as many characters as you've specified after matching the regex in TIME_PREFIX. In this case, it would start at the second timestamp and look 20 characters ahead, means it should see '10/06/2011,14:59:29' as the timestamp.

Re: Another timestamp question

Greg_LeBlanc — Thu, 06 Oct 2011 19:56:51 GMT

Thanks for the answer, I tried this and no luck!

Re: Another timestamp question

alacercogitatus — Thu, 06 Oct 2011 20:02:42 GMT

Your %Y needs capitalized to match the 4 digit year.

Re: Another timestamp question

jbsplunk — Thu, 06 Oct 2011 20:08:22 GMT

corrected. thanks!

Re: Another timestamp question

alacercogitatus — Thu, 06 Oct 2011 20:21:21 GMT

Yep, only on an install that supports indexing (which light and universal disable). http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Forwardercapabilities

Re: Another timestamp question

alacercogitatus — Thu, 06 Oct 2011 20:21:22 GMT

I've never really used the TIME_PREFIX, however from what I'm reading, the parser will try and find the TIME_PREFIX pattern first (which should match 2851,10/06/2011,18:59:29,) and then look the next 20 characters (from MAX_TIMESTAMP_LOOKAHEAD), which should then match 10/06/2011,14:59:29, which includes the comma. And since there is a comma, its not a "true" timestamp, as you will only have either a date or a time in the string. So it is never actually finding the second timestamp. There is also no need to escape the colons.

Try this:

TIME_FORMAT = %m/%d/%Y,%H:%M:%S

TIME_PREFIX = \d{4},\d{2}/\d{2}/\d{4},\d{2}:\d{2}:\d{2},

More details here

Re: Another timestamp question

jbsplunk — Mon, 28 Sep 2020 09:57:29 GMT

The extra character in MAX_TIMESTAMP_LOOKAHEAD shouldn't stop this from working. In fact, sometimes to be safe I'll intentionally add an extra digit if there things like a space or comma. I haven't noticed any strange behavior because of this, but the approach mentioned here is a valid.

Re: Another timestamp question

alacercogitatus — Mon, 28 Sep 2020 09:57:32 GMT

I guess I was a little unclear on which comma. I meant the one in the middle of the date and time, not the one on the end.

10/06/2011*,*14:59:29,

Re: Another timestamp question

Greg_LeBlanc — Thu, 06 Oct 2011 20:41:49 GMT

Thanks! I will move the props.conf to our Heavy Forwarder.