Getting Data In
Highlighted

How to parse out milliseconds from timestamp in log file?

New Member
BREAK_ONLY_BEFORE=\d{7}
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=true
TIME_FORMAT=%3N
TIME_PREFIX=\d{7}

Trying to parse out the millisecond timestamp from this log file, 9281736 :

  9281736 : COUNT IN 1003
Tx:   01 04 00 71 00 02 21 d0                              ...q..!.
Rx:   01 04 04 00 08 0a 28 7c f8    
  9282136 : COUNT IN 1003
Tx:   01 04 00 c9 00 02 a1 f5                              ........
Rx:   01 04 04 00 08 00 00 7a 46 

I suspect my TIME_FORMAT is wrong, because it breaks up events correctly with regex \d{7}.
But according to the error message it doesn't look for the timestamp in the correct spot. Or is it?

Could not use strptime to parse timestamp from ": COUNT 1003\n ......
Make sure a prefix pattern is
specified if the events don`t begin
with a timestamp.

Failed to parse timestamp. Defaulting to file modtime.

Any suggestion would be much appreciated!

0 Karma
Highlighted

Re: How to parse out milliseconds from timestamp in log file?

SplunkTrust
SplunkTrust

Remove the TIME_PREFIX statement. That tells Splunk to start looking for a timestamp after finding 7 digits.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: How to parse out milliseconds from timestamp in log file?

New Member

That makes the timestamp appear in the in the error message. No changes appart from that in Data preview.

Could not use strptime to parse
timestamp from "9281736 : COUNT 1003n
...... Make sure a prefix pattern is
specified if the events don`t begin
with a timestamp. Failed to parse
timestamp. Defaulting to file modtime.

0 Karma
Highlighted

Re: How to parse out milliseconds from timestamp in log file?

SplunkTrust
SplunkTrust

At least now Splunk is looking in the right place for the timestamp. Did you insert the asterisks or are they part of the data?

Tell us more about your data. What does it represent? Milliseconds since when?

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: How to parse out milliseconds from timestamp in log file?

New Member

No, the asterisks is not part of the data. See the logfile entries in the above.

The logfile represent modbus communications between a master and a slave. The milliseconds is since last restart of the master node. I want to trend out timeout periods of the communications and look at what happens just before a period of timeout/no RX.

I`m also interested in other ways to do this. The logfiles are aprox 250 mb

0 Karma
Highlighted

Re: How to parse out milliseconds from timestamp in log file?

SplunkTrust
SplunkTrust

Without a fixed reference point, I don't see how Splunk can convert your milliseconds into timestamps. Perhaps someone else will have a suggestion.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: How to parse out milliseconds from timestamp in log file?

New Member

It knows the first entry. Should`t that be OK?

0 Karma
Highlighted

Re: How to parse out milliseconds from timestamp in log file?

SplunkTrust
SplunkTrust

Each event is processed independently and is unaware of what was processed previously.
Maybe you could write a script to process your log files and convert the milliseconds into timestamps that Splunk can process.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: How to parse out milliseconds from timestamp in log file?

New Member

I could convert milliseconds to hours, minuttes and seconds.. But would that make any difference? Or do I need the date also to give Splunk sufficient reference?

I would assume Splunk sets the reference point at 0 in a 7 digit milisecond timestamp..

0 Karma
Highlighted

Re: How to parse out milliseconds from timestamp in log file?

SplunkTrust
SplunkTrust

Given only a time, Splunk will use the current date.

Assuming you could get Splunk to understand a 7 digit millisecond timestamp (I failed to do so), it would likely use zero as its reference point and you would end up with a timestamp in early 1970.

---
If this reply helps you, an upvote would be appreciated.
0 Karma