topic Re: Splunk Addon data-inputs duplicate events in All Apps and Add-ons

Splunk Addon data-inputs duplicate events

jawaharas — Sun, 15 Mar 2020 06:04:54 GMT

Hello,

I have created an custom add-on to pull events from 'Teachworks' API. But, as per my config (refer screenshot), duplicate records are created whenever the API call try pull the new events.

Example:
Run 1: 5 records available. 5 records pulled into Splunk
Run 2: 5 records available. 0 records pulled into Splunk
Run 3: 6 records available. 6 records pulled into Splunk

I expect only 1 record (new entry) to be pulled into Splunk during 'Run 3', not all 6 records. Any assistance will be helpful.

Re: Splunk Addon data-inputs duplicate events

jawaharas — Sun, 22 Mar 2020 02:40:49 GMT

The issue was due to missing parameter in the REST API URL.

When I add the checkpoint parameter in the REST API call (as below), I don't see duplicate events being pulled.
https://api.teachworks.com/v1/lessons?from_date[gt]>01-01-2020

Reference:
https://docs.splunk.com/Documentation/AddonBuilder/3.0.1/UserGuide/ConfigureDataCollection#Use_checkpoints_for_incremental_data_collection

Re: Splunk Addon data-inputs duplicate events

garunkumar0506 — Wed, 09 Feb 2022 18:33:21 GMT

Hi jawaharas,

We are facing the same issue with different app which we are using. using rest API call we are trying to pull the data but whenever the API calls the entire file content will be get read and indexed. Due to we are indexed lots of duplicate data.

Can you help us with how and where you set the check point to avoid duplicating the data to index ?

Thanks