I have Splunk instance where I configure Data Inputs as "REST API input for polling data from RESTful endpoints". I have almost around 20+ endpoints and where I am pulling data in JSON format and loading in single index.
However, each time any reports or search query runs it will double same data again, like very first fetch brings 5 values and subsequent fetch will bring another 5 and so on and keep increasing.
Now in my dashboards and reports I kind of landed into problem of duplicate data. How I should avoid it? So for very unusual work around I increased interval from 1 min to 1 months, which helps me to avoid data duplication. However, I cannot have stale data for month...I can still survive with 1 day interval, but not with 1 month.
Is there any way in Splunk where I can keep my REST API Call tidy(avoid duplicates) ... to make my dashboards and reports on the fly?
Here is snippet of my inputs.conf file for REST API: