Hi Everyone, I am sending data to snowflake. I send data every 15 mins. In DB Connect output search I make sure I add extra 5 mins for additional data if there is a data loss. During peak hours, when data is more than 2000 rows, I see data missing in snowflake. I do upsert to dedup the duplicates as our TransactionId is unique. Our query generates time in the format:05/06/2026 12:01:57.786 AM EDT but I convert it to 2026-05-06 00:01:57.786 -0400 to accommodate snowflake time requirement. Here is the transformation I use to accommodate it: | eval DateTime_ET_clean = DateTime_ET | eval DateTime_ET_clean = replace(DateTime_ET_clean, " EDT$", " -04:00") | eval DateTime_ET_clean = replace(DateTime_ET_clean, " EST$", " -05:00") | eval DateTime_ET_epoch = strptime(DateTime_ET_clean, "%m/%d/%Y %I:%M:%S.%3N %p %z") | eval DateTime_ET_SF = strftime(DateTime_ET_epoch, "%Y-%m-%d %H:%M:%S.%3N %z") After all these, I still see 20-30% data loss. We are not sure how to make sure all the data is propagated. Out snowflake warehouse is XS in size.
... View more