Splunk Addon data-inputs duplicate events

[jawaharas]

Hello,

I have created an custom add-on to pull events from 'Teachworks' API. But, as per my config (refer screenshot), duplicate records are created whenever the API call try pull the new events.

Example:
Run 1: 5 records available. 5 records pulled into Splunk
Run 2: 5 records available. 0 records pulled into Splunk
Run 3: 6 records available. 6 records pulled into Splunk

I expect only 1 record (new entry) to be pulled into Splunk during 'Run 3', not all 6 records. Any assistance will be helpful.

[jawaharas · Accepted solution]

The issue was due to missing parameter in the REST API URL.

When I add the checkpoint parameter in the REST API call (as below), I don't see duplicate events being pulled.
https://api.teachworks.com/v1/lessons?from_date[gt]>01-01-2020

Reference:
https://docs.splunk.com/Documentation/AddonBuilder/3.0.1/UserGuide/ConfigureDataCollection#Use_check...

[garunkumar0506]

Hi jawaharas,

We are facing the same issue with different app which we are using. using rest API call we are trying to pull the data but whenever the API calls the entire file content will be get read and indexed. Due to we are indexed lots of duplicate data. 

Can you help us with how and where you set the check point to avoid duplicating the data to index ?

Thanks