I have some passive dns data that has time stamps that look like this in JSON logs:
{"timestamp":"2021-10-21 16:31:01","timestamp_s":1634833861,"timestamp_ms":973448,
So it has first conventional time stamp and then a full seconds based Unix Epoch Time Stamp in seconds followed by:
timestamp_ms":990877
This has the millsecs of the time only (actually microseconds). The more convention time would have been:
timestamp_s":1634834347.990877
I have not been able to get the time to include the millisec value included so far. I am using a TIME_PREFIX that should skip the conventional timestamp. Most recently, I used SEDCMD to get the time stamp to look more normal for epoch time --- timestamp_s":1634834347.990877, but maybe the SEDCMD only happens after the time stamp is determined.
I have used similar to for this.
TIME_PREFIX=timestamp_s":
TIME_FORMAT= %s.%6N
Any help appreciated !
TIME_PREFIX = timestamp_s\x22:
TIME_FORMAT = %s,\"timestamp_ms\":%6N
Hi jimmy ,
So i think just index the data using first time stamp in temp index then write a spl to extract this info and push the new data inside your desired index .
you can also set retention time 2-3 weeks in temp index
Thanks for the thought. A possible solution. I'll update.
I had hoped I could just get strptime to skip around the stuff between the secs part of the epoch time and the ms part, TIME_FORMAT = %s,\"timestamp_ms\":%6N, but that did not work -- You can put slashes colon and other punctuation in there but I guess no strings.
I also tried fixing the time stamp to a pure seconds and ms part doing a rewrite with SEDCMD, that worked to re-write the record, but did not fix the time stamp which is determined before.
I don't think I have previously had a post up for this long and gotten no answers. PLEASE !
Please help ! I normall get help after posting. This is an important issue for us.