topic Modular Input Checkpoint in Getting Data In

Modular Input Checkpoint

sanjay_shrestha — Wed, 04 Nov 2015 22:02:11 GMT

I am writing a modular input and the script pulls list of the records in each interval when it runs.

e.g.

Name       Address
Joe         Ankeny
Bob        Clive

I do get duplicate events as I have not implemented Checkpoint yet. Since the script would bring all rows every time, do I need to save this every single row in checkpoint file and run through verification if row exists in the file or not?

Thanks,
Sanjay

Re: Modular Input Checkpoint

MuS — Wed, 04 Nov 2015 22:16:51 GMT

Hi sanjay.shrestha,

if your script pulls in all rows on each run, your approach sounds good.

It would be easier if you could use some timestamp to get the data in; in this case you can safe the last time the script ran in the checkpoint and use this last time stamp in the next script run.
I have some modular inputs doing exactly this.

Hope this helps ...

cheers, MuS

Re: Modular Input Checkpoint

sanjay_shrestha — Thu, 05 Nov 2015 02:09:04 GMT

Thanks Michael. I will implement your advise and let you know.