topic Re: How to parse out milliseconds from timestamp in log file? in Getting Data In

How to parse out milliseconds from timestamp in log file?

nitrogaute — Tue, 28 Oct 2014 11:04:51 GMT

BREAK_ONLY_BEFORE=\d{7}
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=true
TIME_FORMAT=%3N
TIME_PREFIX=\d{7}

Trying to parse out the millisecond timestamp from this log file, 9281736 :

  9281736 : COUNT IN 1003
Tx:   01 04 00 71 00 02 21 d0                              ...q..!.
Rx:   01 04 04 00 08 0a 28 7c f8    
  9282136 : COUNT IN 1003
Tx:   01 04 00 c9 00 02 a1 f5                              ........
Rx:   01 04 04 00 08 00 00 7a 46

I suspect my TIME_FORMAT is wrong, because it breaks up events correctly with regex \d{7}.
But according to the error message it doesn't look for the timestamp in the correct spot. Or is it?

Could not use strptime to parse timestamp from ": COUNT 1003\n ......
Make sure a prefix pattern is
specified if the events don`t begin
with a timestamp.

Failed to parse timestamp. Defaulting to file modtime.

Any suggestion would be much appreciated!

Re: How to parse out milliseconds from timestamp in log file?

richgalloway — Tue, 28 Oct 2014 11:47:46 GMT

Remove the TIME_PREFIX statement. That tells Splunk to start looking for a timestamp after finding 7 digits.

Re: How to parse out milliseconds from timestamp in log file?

nitrogaute — Tue, 28 Oct 2014 11:52:31 GMT

That makes the timestamp appear in the in the error message. No changes appart from that in Data preview.

Could not use strptime to parse
timestamp from "9281736 : COUNT 1003n
...... Make sure a prefix pattern is
specified if the events don`t begin
with a timestamp. Failed to parse
timestamp. Defaulting to file modtime.

Re: How to parse out milliseconds from timestamp in log file?

richgalloway — Tue, 28 Oct 2014 12:20:30 GMT

At least now Splunk is looking in the right place for the timestamp. Did you insert the asterisks or are they part of the data?

Tell us more about your data. What does it represent? Milliseconds since when?

Re: How to parse out milliseconds from timestamp in log file?

nitrogaute — Tue, 28 Oct 2014 12:28:17 GMT

No, the asterisks is not part of the data. See the logfile entries in the above.

The logfile represent modbus communications between a master and a slave. The milliseconds is since last restart of the master node. I want to trend out timeout periods of the communications and look at what happens just before a period of timeout/no RX.

I`m also interested in other ways to do this. The logfiles are aprox 250 mb

Re: How to parse out milliseconds from timestamp in log file?

richgalloway — Tue, 28 Oct 2014 12:35:21 GMT

Without a fixed reference point, I don't see how Splunk can convert your milliseconds into timestamps. Perhaps someone else will have a suggestion.

Re: How to parse out milliseconds from timestamp in log file?

nitrogaute — Tue, 28 Oct 2014 12:38:31 GMT

It knows the first entry. Should`t that be OK?

Re: How to parse out milliseconds from timestamp in log file?

richgalloway — Tue, 28 Oct 2014 12:46:12 GMT

Each event is processed independently and is unaware of what was processed previously.
Maybe you could write a script to process your log files and convert the milliseconds into timestamps that Splunk can process.

Re: How to parse out milliseconds from timestamp in log file?

nitrogaute — Tue, 28 Oct 2014 12:52:02 GMT

I could convert milliseconds to hours, minuttes and seconds.. But would that make any difference? Or do I need the date also to give Splunk sufficient reference?

I would assume Splunk sets the reference point at 0 in a 7 digit milisecond timestamp..

Re: How to parse out milliseconds from timestamp in log file?

richgalloway — Tue, 28 Oct 2014 12:57:22 GMT

Given only a time, Splunk will use the current date.

Assuming you could get Splunk to understand a 7 digit millisecond timestamp (I failed to do so), it would likely use zero as its reference point and you would end up with a timestamp in early 1970.

Re: How to parse out milliseconds from timestamp in log file?

nitrogaute — Tue, 28 Oct 2014 13:00:59 GMT

Hmm, I see youre point. But I still think its strange that Splunk does not handle relative time...