topic Re: Epoch Time - Time Stamp Assignment with Millisecs seperate in JSON in Getting Data In

Epoch Time - Time Stamp Assignment with Millisecs seperate in JSON

jimdiconectiv — Thu, 18 Nov 2021 19:11:19 GMT

I have some passive dns data that has time stamps that look like this in JSON logs:

{"timestamp":"2021-10-21 16:31:01","timestamp_s":1634833861,"timestamp_ms":973448,

So it has first conventional time stamp and then a full seconds based Unix Epoch Time Stamp in seconds followed by:

timestamp_ms":990877

This has the millsecs of the time only (actually microseconds). The more convention time would have been:

timestamp_s":1634834347.990877

I have not been able to get the time to include the millisec value included so far. I am using a TIME_PREFIX that should skip the conventional timestamp. Most recently, I used SEDCMD to get the time stamp to look more normal for epoch time --- timestamp_s":1634834347.990877, but maybe the SEDCMD only happens after the time stamp is determined.

I have used similar to for this.

TIME_PREFIX=timestamp_s":
TIME_FORMAT= %s.%6N

Any help appreciated !

Re: Epoch Time - Time Stamp Assignment with Millisecs seperate in JSON

jimdiconectiv — Mon, 29 Nov 2021 19:22:58 GMT

Please help ! I normall get help after posting. This is an important issue for us.

Re: Epoch Time - Time Stamp Assignment with Millisecs seperate in JSON

jimdiconectiv — Wed, 01 Dec 2021 20:13:36 GMT

I don't think I have previously had a post up for this long and gotten no answers. PLEASE !

Re: Epoch Time - Time Stamp Assignment with Millisecs seperate in JSON

Siddharth — Wed, 01 Dec 2021 21:40:56 GMT

Hi jimmy ,

So i think just index the data using first time stamp in temp index then write a spl to extract this info and push the new data inside your desired index .

you can also set retention time 2-3 weeks in temp index

Re: Epoch Time - Time Stamp Assignment with Millisecs seperate in JSON

jimdiconectiv — Thu, 02 Dec 2021 17:09:26 GMT

Thanks for the thought. A possible solution. I'll update.

I had hoped I could just get strptime to skip around the stuff between the secs part of the epoch time and the ms part, TIME_FORMAT = %s,\"timestamp_ms\":%6N, but that did not work -- You can put slashes colon and other punctuation in there but I guess no strings.

I also tried fixing the time stamp to a pure seconds and ms part doing a rewrite with SEDCMD, that worked to re-write the record, but did not fix the time stamp which is determined before.

Re: Epoch Time - Time Stamp Assignment with Millisecs seperate in JSON

johnhuang — Thu, 02 Dec 2021 22:26:38 GMT

TIME_PREFIX = timestamp_s\x22:
TIME_FORMAT = %s,\"timestamp_ms\":%6N