topic Re: Time/date extraction from our log in Splunk Search

Time/date extraction from our log

peterbrown05 — Mon, 05 Dec 2011 10:53:05 GMT

Hi
Im really struggling to extract the time/date data from our logs. Ive read some of the other topics/docs on doing this, but just can't seem to get it to work. The auto extract field discovery also doesnt work;

or log data looks like;
4972 ClientApp 5108 0 10 Client-10 18/11/2011 23:57:02:19+20 Scheduler2 No tick violation in last period of '00:00:10:00'.

so the time date of this specific log message would be "18/11/2011 23:57:02:19+20" where the +20 denotes the number of microseconds through the millisecond. ie; "dd/mm/yyyy hh:mm:ss:ms+us"

Anyone know how to regex this so that splunk can index it properly?

cheers
p

Re: Time/date extraction from our log

Ayn — Mon, 05 Dec 2011 12:29:47 GMT

How is it not working? Time is completely wrong, or almost correct but not catching the milli/microseconds?

Re: Time/date extraction from our log

kristian_kolb — Mon, 05 Dec 2011 20:00:59 GMT

I love precision just as much as the next guy, but do you NEED the microseconds? Or would you be happy enough with the milliseconds? Or seconds?

If I understand your time format correctly, it really says 23:57:02:019020

That's 19 milliseconds and another 20 microseconds. So there are no leading zeros in your timestamp format, for either ms or us, right?

If that is the case, I'd either

a) be happy with SECONDS. Skip everything after %H:%M:%S in your TIME_FORMAT
or
b) change the log format at the source. Leading zeros... you gotta have those.

UPDATE:

If you do not really need the events to be indexed at the correct millisecond, there is no need to try to fix any conversion at all (i.e. frame_no * 40 = ms). If I understood you correctly, you want to have the frame number as part of the timestamp, correct?

Would it then be possible to pretend that the frame number is actually the fraction of a second? Ending the TIME_FORMAT= with ...%M:%S:%2N+ lets Splunk believe that the frame number value is in hundredths of a second.

Looking at the events, especially in a timeline format, would obviously be wrong, since no events would ever come in between .25 and .99 of any given second - but you'd know this. Precision would still be pretty good for most intents and purposes, and you'd be indexing events on year-month-day-hour-minute-second-frame.

UPDATE2:

The 'solution' suggested above require that frames are written with leading zeros, i.e. the first frame is logged as '01' and not '1'. If not, there will be no distinction between frames 1 and 10, and frames 2 and 20. The rest should be fine. Forgot to mention that in my previous post. Sorry.

Maybe this approach is too much of cutting corners, but please let us know what you think.

hth,

Kristian

Re: Time/date extraction from our log

dwaddle — Mon, 05 Dec 2011 22:32:37 GMT

I'm not sure Splunk can deal with both microseconds and milliseconds.

Re: Time/date extraction from our log

peterbrown05 — Tue, 06 Dec 2011 08:02:17 GMT

sorry, maybe i over simplified my original question. the ms are actually "frames" as we work in broadcast. so a frame (in PAL regions) is 40ms. (0 to 24frames in a second) so thats actually 19x40ms. To us the frames are important but we could ignore anything after the +.

We wouldnt want to convert the frames to milliseconds, more just that it should log on a specific frame in the example above on frame 19.

sorry for causing confusion,
peteB

Re: Time/date extraction from our log

Ayn — Tue, 06 Dec 2011 08:41:07 GMT

OK, thanks for the clarification. So - what works and what does not work?

Re: Time/date extraction from our log

kristian_kolb — Tue, 06 Dec 2011 21:50:43 GMT

see update above.

Re: Time/date extraction from our log

peterbrown05 — Wed, 07 Dec 2011 09:21:54 GMT

yeah; I think this is along the right lines. however, its not possible for us to change the log format as our software is installed on many customer sites which means it is impossible for us to upgrade across the board.

We are evaluating splunk to help us mine the data that is already being generated. Perhaps then there is a way (using some regex magic?) to index the data using Frames x 40ms, but then just displaying the "field" as the hh:mm:ss+XX.

Re: Time/date extraction from our log

peterbrown05 — Wed, 07 Dec 2011 09:21:59 GMT

It certainly is useful for us to be able to see "what happened" at a specific frame; eg, video should start playing, router switches program to output, channel logo is displayed etc - all of which may be logged out on different systems into different log files; but all machines are synchronised to the same "house" time.