Getting Data In

How can I avoid duplication of data pulled by REST API?

New Member

I have Splunk instance where I configure Data Inputs as "REST API input for polling data from RESTful endpoints". I have almost around 20+ endpoints and where I am pulling data in JSON format and loading in single index.

However, each time any reports or search query runs it will double same data again, like very first fetch brings 5 values and subsequent fetch will bring another 5 and so on and keep increasing.

Now in my dashboards and reports I kind of landed into problem of duplicate data. How I should avoid it? So for very unusual work around I increased interval from 1 min to 1 months, which helps me to avoid data duplication. However, I cannot have stale data for month...I can still survive with 1 day interval, but not with 1 month.

Is there any way in Splunk where I can keep my REST API Call tidy(avoid duplicates) ... to make my dashboards and reports on the fly?

Here is snippet of my inputs.conf file for REST API:

auth_password = ccccc
auth_type = basic
auth_user = vvvvvvv
endpoint =
host = slrestdata
http_method = GET
index = sldata
index_error_response_codes = 0
response_type = json
sequential_mode = 0
sourcetype = _json
streaming_request = 0
polling_interval = 2592000
0 Karma

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.