Epoch Time - Time Stamp Assignment with Millisecs seperate in JSON

[jimdiconectiv]

I have some passive dns data that has time stamps that look like this in JSON logs:

{"timestamp":"2021-10-21 16:31:01","timestamp_s":1634833861,"timestamp_ms":973448, 

So it has first conventional time stamp and then a full seconds based Unix Epoch Time Stamp in seconds followed by:

timestamp_ms":990877

This has the millsecs of the time only (actually microseconds).  The more convention time would have been:

timestamp_s":1634834347.990877 

I have not been able to get the time to include the millisec value included so far.  I am using a TIME_PREFIX that should skip the conventional timestamp.   Most recently, I used SEDCMD to get the time stamp to look more normal for epoch time --- timestamp_s":1634834347.990877,  but maybe the SEDCMD only happens after the time stamp is determined.

I have used similar to for this.

TIME_PREFIX=timestamp_s":
TIME_FORMAT= %s.%6N

Any help appreciated ! 

[johnhuang]

TIME_PREFIX = timestamp_s\x22:
TIME_FORMAT = %s,\"timestamp_ms\":%6N

[Siddharth]

Hi jimmy ,

So i think just index the data using first time stamp in temp index then write a spl to extract this info and push the new data inside your desired index .

you can also set retention time 2-3 weeks in temp index

[jimdiconectiv]

Thanks for the thought.  A possible solution.  I'll update.  

I had hoped I could just get strptime to skip around the stuff between the secs part of the epoch time and the ms part, TIME_FORMAT = %s,\"timestamp_ms\":%6N,  but that did not work -- You can put slashes colon and other punctuation in there but I guess no strings. 

I also tried fixing the time stamp to a pure seconds and ms part doing a rewrite with SEDCMD, that worked to re-write the record,  but did not fix the time stamp which is determined before. 

[jimdiconectiv]

I don't think I have previously had a post up for this long and gotten no answers.   PLEASE ! 

[jimdiconectiv]

Please help !   I normall get help after posting.  This is an important issue for us.