Getting Data In

How can I avoid duplication of data pulled by REST API?

niravhjoshi
New Member

I have Splunk instance where I configure Data Inputs as "REST API input for polling data from RESTful endpoints". I have almost around 20+ endpoints and where I am pulling data in JSON format and loading in single index.

However, each time any reports or search query runs it will double same data again, like very first fetch brings 5 values and subsequent fetch will bring another 5 and so on and keep increasing.

Now in my dashboards and reports I kind of landed into problem of duplicate data. How I should avoid it? So for very unusual work around I increased interval from 1 min to 1 months, which helps me to avoid data duplication. However, I cannot have stale data for month...I can still survive with 1 day interval, but not with 1 month.

Is there any way in Splunk where I can keep my REST API Call tidy(avoid duplicates) ... to make my dashboards and reports on the fly?

Here is snippet of my inputs.conf file for REST API:

[rest://rst_sl_get_version]
auth_password = ccccc
auth_type = basic
auth_user = vvvvvvv
endpoint = https://api.xx.com/rest/v3/xx_version
host = slrestdata
http_method = GET
index = sldata
index_error_response_codes = 0
response_type = json
sequential_mode = 0
sourcetype = _json
streaming_request = 0
polling_interval = 2592000
0 Karma

valiquet
Contributor
0 Karma
Get Updates on the Splunk Community!

AppDynamics Summer Webinars

This summer, our mighty AppDynamics team is cooking up some delicious content on YouTube Live to satiate your ...

SOCin’ it to you at Splunk University

Splunk University is expanding its instructor-led learning portfolio with dedicated Security tracks at .conf25 ...

Credit Card Data Protection & PCI Compliance with Splunk Edge Processor

Organizations handling credit card transactions know that PCI DSS compliance is both critical and complex. The ...