All Apps and Add-ons

Splunk Addon data-inputs duplicate events

jawaharas
Motivator

Hello,

I have created an custom add-on to pull events from 'Teachworks' API. But, as per my config (refer screenshot), duplicate records are created whenever the API call try pull the new events.

Example:
Run 1: 5 records available. 5 records pulled into Splunk
Run 2: 5 records available. 0 records pulled into Splunk
Run 3: 6 records available. 6 records pulled into Splunk

I expect only 1 record (new entry) to be pulled into Splunk during 'Run 3', not all 6 records. Any assistance will be helpful.

alt text

0 Karma
1 Solution

jawaharas
Motivator

The issue was due to missing parameter in the REST API URL.

When I add the checkpoint parameter in the REST API call (as below), I don't see duplicate events being pulled.
https://api.teachworks.com/v1/lessons?from_date[gt]>01-01-2020

Reference:
https://docs.splunk.com/Documentation/AddonBuilder/3.0.1/UserGuide/ConfigureDataCollection#Use_check...

View solution in original post

0 Karma

jawaharas
Motivator

The issue was due to missing parameter in the REST API URL.

When I add the checkpoint parameter in the REST API call (as below), I don't see duplicate events being pulled.
https://api.teachworks.com/v1/lessons?from_date[gt]>01-01-2020

Reference:
https://docs.splunk.com/Documentation/AddonBuilder/3.0.1/UserGuide/ConfigureDataCollection#Use_check...

0 Karma

garunkumar0506
New Member

Hi jawaharas,

We are facing the same issue with different app which we are using. using rest API call we are trying to pull the data but whenever the API calls the entire file content will be get read and indexed. Due to we are indexed lots of duplicate data. 

Can you help us with how and where you set the check point to avoid duplicating the data to index ?

 

Thanks

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...