Getting Data In

Modular Input Checkpoint

sanjay_shrestha
Contributor

I am writing a modular input and the script pulls list of the records in each interval when it runs.

e.g.

Name       Address
Joe         Ankeny
Bob        Clive

I do get duplicate events as I have not implemented Checkpoint yet. Since the script would bring all rows every time, do I need to save this every single row in checkpoint file and run through verification if row exists in the file or not?

Thanks,
Sanjay

0 Karma
1 Solution

MuS
Legend

Hi sanjay.shrestha,

if your script pulls in all rows on each run, your approach sounds good.

It would be easier if you could use some timestamp to get the data in; in this case you can safe the last time the script ran in the checkpoint and use this last time stamp in the next script run.
I have some modular inputs doing exactly this.

Hope this helps ...

cheers, MuS

View solution in original post

0 Karma

MuS
Legend

Hi sanjay.shrestha,

if your script pulls in all rows on each run, your approach sounds good.

It would be easier if you could use some timestamp to get the data in; in this case you can safe the last time the script ran in the checkpoint and use this last time stamp in the next script run.
I have some modular inputs doing exactly this.

Hope this helps ...

cheers, MuS

0 Karma

sanjay_shrestha
Contributor

Thanks Michael. I will implement your advise and let you know.

0 Karma
Get Updates on the Splunk Community!

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...