topic Re: Stop times like '0:20:00' being read as 8pm in Getting Data In

Stop times like '0:20:00' being read as 8pm

parallaxed — Fri, 23 Apr 2010 19:24:03 GMT

Splunk always seems to get this wrong. I have the following in a vain effort to correct this

TIME_PREFIX=^

TIME_FORMAT=%D%t%T

Didn't really do anything for the situation. Wondering if there's some other config I can try?

Re: Stop times like '0:20:00' being read as 8pm

Mick — Fri, 23 Apr 2010 21:49:57 GMT

There's many other possible configs you can try, but trying to guess the exact format of the entire timestamp is beyond my capabilities. When you ask a question that'd dependant on the format of an event, why not paste in an example so we can see what you're talking about? If there's sensitve data in there just anonymize it with x's or something.

This is the page you want to read first - Enhanced strptime() support and then here

Your setting - TIME_FORMAT=%D%t%T

%D - The date as %m / %d / %y.
%t - white space
%T - The time as %H : %M : %S.

Where did you get these values?

How about - TIME_FORMAT=%H:%M:%s

%H = The hour (24-hour clock)
%M = The minute [00,59]
%S = The seconds [00,60]

Note the ':' in there as well, you're specifying the exact format of the timestamp so you have to include everything that is contained within it.

Re: Stop times like '0:20:00' being read as 8pm

gkanapathy — Sun, 25 Apr 2010 03:59:27 GMT

In addition to Mick's answer (which is that you should use the correct TIME_FORMAT according to what is in your data) you may also need to look into the MAX_TIMESTAMP_LOOKAHEAD setting, which will control how far into a line Splunk will keep looking for a timestamp. Simply setting the TIME_PREFIX just tells Splunk to start looking for times is sees the TIME_PREFIX, but it could be anywhere after the prefix.

If you provide what an actual line of data looks like, perhaps we can help. If the TIME_FORMAT can't be found, Splunk rather aggressively attempts to derive some timestamp from your data, as a time is always required to write an event into the Splunk index.

Re: Stop times like '0:20:00' being read as 8pm

parallaxed — Fri, 21 May 2010 23:21:24 GMT

The event line looks like the following:

05/20/2010 0:20:00,someData,1274329200,60,0,2,someData,8256952.0,3913828.0,4007580.0,50.0

^ the timestamp is recognized as 8pm.

The only exception that I can see here is that there are three lots of whitespace between %Y and %H, shouldn't %t cover multiple spaces anyhow? It seems the '0:' following %t is being ignored.

Re: Stop times like '0:20:00' being read as 8pm

parallaxed — Fri, 21 May 2010 23:22:06 GMT

Seems whitespace doesn't work, hope that's clear though...

Re: Stop times like '0:20:00' being read as 8pm

parallaxed — Fri, 21 May 2010 23:30:40 GMT

Further to the comment above, I should add that I've tested with %D%t%t%t%T and more specific strptime formats, none of them seem to work with this kind of event (i.e. missing a leading 0 off the %T). Is this a bug?

Re: Stop times like '0:20:00' being read as 8pm

parallaxed — Fri, 21 May 2010 23:38:51 GMT

Additionally, as a consequence of the above, the times 0:01:00 through 0:23:00 are also affected (interpreted as 24 hour times exc. leading 0). 0:24:00 is recognized correctly as 12:24AM, example line:

05/20/2010 0:24:00,someData,1274329440,60,0,2,8256952.0,3913828.0,4007580.0,50.0

Re: Stop times like '0:20:00' being read as 8pm

gkanapathy — Sat, 22 May 2010 07:51:39 GMT

Could you try using %H:%M:%S instead of %T? I have never used %T.

Re: Stop times like '0:20:00' being read as 8pm

parallaxed — Tue, 25 May 2010 14:38:03 GMT

I've tried both, neither work.

Re: Stop times like '0:20:00' being read as 8pm

gkanapathy — Tue, 25 May 2010 19:43:22 GMT

i'd say pretty much a bug. it might be possible to work around it with a custom datetime.xml setting.

Re: Stop times like '0:20:00' being read as 8pm

parallaxed — Thu, 27 May 2010 18:29:05 GMT

This is due to the fact that default etc/datetime.xml regex is not greedy enough:

Adding an additional clause fixed this: