I fixed the title. I didn’t get you quite well. The amount ”Enter an integer between 1 and 10000000." Is this the limit of one time fetch. If yes then the next time (depending on the frequency we set) it can fetch the same rows, this way it’d never finish the full list (specially in case of batch input). Couple of things regarding the database I am dealing with, it is increasing with rate of almost more than half a million new entries every day. My two main options are batch input Vs rising column. Let's discuss both of them.
Followup question:
A) Batch Input
A1: One problem is that if we pull 10K every 5 minutes, as proposed above. It will take a long time to catch-up and with the amount of logs the db is increasing (half a million/per day) this seems even more challenging.
A2: Even it completes at some point the the data would start duplicating.
B) Rising Column Input
B1: When I run the same exact query in Rising column, it takes about 7/10 minutes before throwing error like below:
Invalid Query
External search command 'dbxquery' returned error code 1. Script output = "RuntimeError: Failed to run query: "SELECT * FROM (SELECT * from db . apps ) t", params: "None", caused by: Exception(" java.sql.SQLException: Incorrect key file for table 'C:\Windows\TEMP\#sql22c_2b711e_2.MYI'; try to repair it.",). "
B2: Let's say we solve the above issue, I am not sure how the limit of 10 million would affect here with the rate it is growing.
... View more
