https://community.splunk.com/t5/Getting-Data-In/How-can-I-avoid-duplication-of-data-pulled-by-REST-API/m-p/298323#M56445 <P>I have Splunk instance where I configure Data Inputs as "REST API input for polling data from RESTful endpoints". I have almost around 20+ endpoints and where I am pulling data in JSON format and loading in single index.</P> <P>However, each time any reports or search query runs it will double same data again, like very first fetch brings 5 values and subsequent fetch will bring another 5 and so on and keep increasing.</P> <P>Now in my dashboards and reports I kind of landed into problem of duplicate data. How I should avoid it? So for very unusual work around I increased interval from 1 min to 1 months, which helps me to avoid data duplication. However, I cannot have stale data for month...I can still survive with 1 day interval, but not with 1 month.</P> <P><STRONG>Is there any way in Splunk where I can keep my REST API Call tidy(avoid duplicates) ... to make my dashboards and reports on the fly?</STRONG></P> <P>Here is snippet of my inputs.conf file for REST API:</P> <PRE><CODE>[rest://rst_sl_get_version] auth_password = ccccc auth_type = basic auth_user = vvvvvvv endpoint = <A href="https://api.xx.com/rest/v3/xx_version" target="test_blank">https://api.xx.com/rest/v3/xx_version</A> host = slrestdata http_method = GET index = sldata index_error_response_codes = 0 response_type = json sequential_mode = 0 sourcetype = _json streaming_request = 0 polling_interval = 2592000 </CODE></PRE> Wed, 23 Aug 2017 10:12:43 GMT niravhjoshi 2017-08-23T10:12:43Z https://community.splunk.com/t5/Getting-Data-In/How-can-I-avoid-duplication-of-data-pulled-by-REST-API/m-p/298323#M56445 <P>I have Splunk instance where I configure Data Inputs as "REST API input for polling data from RESTful endpoints". I have almost around 20+ endpoints and where I am pulling data in JSON format and loading in single index.</P> <P>However, each time any reports or search query runs it will double same data again, like very first fetch brings 5 values and subsequent fetch will bring another 5 and so on and keep increasing.</P> <P>Now in my dashboards and reports I kind of landed into problem of duplicate data. How I should avoid it? So for very unusual work around I increased interval from 1 min to 1 months, which helps me to avoid data duplication. However, I cannot have stale data for month...I can still survive with 1 day interval, but not with 1 month.</P> <P><STRONG>Is there any way in Splunk where I can keep my REST API Call tidy(avoid duplicates) ... to make my dashboards and reports on the fly?</STRONG></P> <P>Here is snippet of my inputs.conf file for REST API:</P> <PRE><CODE>[rest://rst_sl_get_version] auth_password = ccccc auth_type = basic auth_user = vvvvvvv endpoint = <A href="https://api.xx.com/rest/v3/xx_version" target="test_blank">https://api.xx.com/rest/v3/xx_version</A> host = slrestdata http_method = GET index = sldata index_error_response_codes = 0 response_type = json sequential_mode = 0 sourcetype = _json streaming_request = 0 polling_interval = 2592000 </CODE></PRE> Wed, 23 Aug 2017 10:12:43 GMT https://community.splunk.com/t5/Getting-Data-In/How-can-I-avoid-duplication-of-data-pulled-by-REST-API/m-p/298323#M56445 niravhjoshi 2017-08-23T10:12:43Z https://community.splunk.com/t5/Getting-Data-In/How-can-I-avoid-duplication-of-data-pulled-by-REST-API/m-p/298324#M56446 <P>Try splunk_server=xxx</P> <P><A href="https://docs.splunk.com/Documentation/Splunk/7.2.1/SearchReference/Rest">https://docs.splunk.com/Documentation/Splunk/7.2.1/SearchReference/Rest</A></P> Fri, 21 Dec 2018 19:39:05 GMT https://community.splunk.com/t5/Getting-Data-In/How-can-I-avoid-duplication-of-data-pulled-by-REST-API/m-p/298324#M56446 valiquet 2018-12-21T19:39:05Z