Getting Data In

How to parse out milliseconds from timestamp in log file?

nitrogaute
New Member
BREAK_ONLY_BEFORE=\d{7}
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=true
TIME_FORMAT=%3N
TIME_PREFIX=\d{7}

Trying to parse out the millisecond timestamp from this log file, 9281736 :

  9281736 : COUNT IN 1003
Tx:   01 04 00 71 00 02 21 d0                              ...q..!.
Rx:   01 04 04 00 08 0a 28 7c f8    
  9282136 : COUNT IN 1003
Tx:   01 04 00 c9 00 02 a1 f5                              ........
Rx:   01 04 04 00 08 00 00 7a 46 

I suspect my TIME_FORMAT is wrong, because it breaks up events correctly with regex \d{7}.
But according to the error message it doesn't look for the timestamp in the correct spot. Or is it?

Could not use strptime to parse timestamp from ": COUNT 1003\n ......
Make sure a prefix pattern is
specified if the events don`t begin
with a timestamp.

Failed to parse timestamp. Defaulting to file modtime.

Any suggestion would be much appreciated!

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Remove the TIME_PREFIX statement. That tells Splunk to start looking for a timestamp after finding 7 digits.

---
If this reply helps you, Karma would be appreciated.
0 Karma

nitrogaute
New Member

That makes the timestamp appear in the in the error message. No changes appart from that in Data preview.

Could not use strptime to parse
timestamp from "9281736 : COUNT 1003n
...... Make sure a prefix pattern is
specified if the events don`t begin
with a timestamp. Failed to parse
timestamp. Defaulting to file modtime.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

At least now Splunk is looking in the right place for the timestamp. Did you insert the asterisks or are they part of the data?

Tell us more about your data. What does it represent? Milliseconds since when?

---
If this reply helps you, Karma would be appreciated.
0 Karma

nitrogaute
New Member

No, the asterisks is not part of the data. See the logfile entries in the above.

The logfile represent modbus communications between a master and a slave. The milliseconds is since last restart of the master node. I want to trend out timeout periods of the communications and look at what happens just before a period of timeout/no RX.

I`m also interested in other ways to do this. The logfiles are aprox 250 mb

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Without a fixed reference point, I don't see how Splunk can convert your milliseconds into timestamps. Perhaps someone else will have a suggestion.

---
If this reply helps you, Karma would be appreciated.
0 Karma

nitrogaute
New Member

It knows the first entry. Should`t that be OK?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Each event is processed independently and is unaware of what was processed previously.
Maybe you could write a script to process your log files and convert the milliseconds into timestamps that Splunk can process.

---
If this reply helps you, Karma would be appreciated.
0 Karma

nitrogaute
New Member

I could convert milliseconds to hours, minuttes and seconds.. But would that make any difference? Or do I need the date also to give Splunk sufficient reference?

I would assume Splunk sets the reference point at 0 in a 7 digit milisecond timestamp..

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Given only a time, Splunk will use the current date.

Assuming you could get Splunk to understand a 7 digit millisecond timestamp (I failed to do so), it would likely use zero as its reference point and you would end up with a timestamp in early 1970.

---
If this reply helps you, Karma would be appreciated.
0 Karma

nitrogaute
New Member

Hmm, I see youre point. But I still think its strange that Splunk does not handle relative time...

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...