I have Splunk instance where I configure Data Inputs as "REST API input for polling data from RESTful endpoints". I have almost around 20+ endpoints and where I am pulling data in JSON format and loading in single index.
However, each time any reports or search query runs it will double same data again, like very first fetch brings 5 values and subsequent fetch will bring another 5 and so on and keep increasing.
Now in my dashboards and reports I kind of landed into problem of duplicate data. How I should avoid it? So for very unusual work around I increased interval from 1 min to 1 months, which helps me to avoid data duplication. However, I cannot have stale data for month...I can still survive with 1 day interval, but not with 1 month.
Is there any way in Splunk where I can keep my REST API Call tidy(avoid duplicates) ... to make my dashboards and reports on the fly?
Here is snippet of my inputs.conf file for REST API:
[rest://rst_sl_get_version]
auth_password = ccccc
auth_type = basic
auth_user = vvvvvvv
endpoint = https://api.xx.com/rest/v3/xx_version
host = slrestdata
http_method = GET
index = sldata
index_error_response_codes = 0
response_type = json
sequential_mode = 0
sourcetype = _json
streaming_request = 0
polling_interval = 2592000
