Getting Data In

Modular Input Checkpoint

sanjay_shrestha
Contributor

I am writing a modular input and the script pulls list of the records in each interval when it runs.

e.g.

Name       Address
Joe         Ankeny
Bob        Clive

I do get duplicate events as I have not implemented Checkpoint yet. Since the script would bring all rows every time, do I need to save this every single row in checkpoint file and run through verification if row exists in the file or not?

Thanks,
Sanjay

0 Karma
1 Solution

MuS
SplunkTrust
SplunkTrust

Hi sanjay.shrestha,

if your script pulls in all rows on each run, your approach sounds good.

It would be easier if you could use some timestamp to get the data in; in this case you can safe the last time the script ran in the checkpoint and use this last time stamp in the next script run.
I have some modular inputs doing exactly this.

Hope this helps ...

cheers, MuS

View solution in original post

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi sanjay.shrestha,

if your script pulls in all rows on each run, your approach sounds good.

It would be easier if you could use some timestamp to get the data in; in this case you can safe the last time the script ran in the checkpoint and use this last time stamp in the next script run.
I have some modular inputs doing exactly this.

Hope this helps ...

cheers, MuS

0 Karma

sanjay_shrestha
Contributor

Thanks Michael. I will implement your advise and let you know.

0 Karma
Get Updates on the Splunk Community!

Take the 2021 Splunk Career Survey for $50 in Amazon Cash

Help us learn about how Splunk has impacted your career by taking the 2021 Splunk Career Survey. Last year’s ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

Observability Newsletter Highlights | March 2023

 March 2023 | Check out the latest and greatestSplunk APM's New Tag Filter ExperienceSplunk APM has updated ...