https://community.splunk.com/t5/Splunk-Dev/How-to-avoid-splunk-from-indexing-duplicate-data-Sending-using/m-p/512808#M9151 <P>Hello everyone!<BR /><BR />I am using a python script to send some data to a splunk instance on my own machine. The code is something like this:</P><P>service = client.connect(host='localhost',port=8089,username='SOMETHING',password='SOMETHING2')<BR />myindex = service.indexes["indexName"]<BR />myindex.submit(jsonData, sourcetype="bobby", host="local")</P><P>&nbsp;</P><P>Is there any way to "tell" Splunk not to index anything duplicated? That is, anything that already exists in the index.</P><P>I know i could mess with the script to avoid sending duplicates, but if splunk could do it, things would be easier.</P><P>Thanks!&nbsp;</P> Thu, 06 Aug 2020 16:38:23 GMT LAcioffi 2020-08-06T16:38:23Z https://community.splunk.com/t5/Splunk-Dev/How-to-avoid-splunk-from-indexing-duplicate-data-Sending-using/m-p/512808#M9151 <P>Hello everyone!<BR /><BR />I am using a python script to send some data to a splunk instance on my own machine. The code is something like this:</P><P>service = client.connect(host='localhost',port=8089,username='SOMETHING',password='SOMETHING2')<BR />myindex = service.indexes["indexName"]<BR />myindex.submit(jsonData, sourcetype="bobby", host="local")</P><P>&nbsp;</P><P>Is there any way to "tell" Splunk not to index anything duplicated? That is, anything that already exists in the index.</P><P>I know i could mess with the script to avoid sending duplicates, but if splunk could do it, things would be easier.</P><P>Thanks!&nbsp;</P> Thu, 06 Aug 2020 16:38:23 GMT https://community.splunk.com/t5/Splunk-Dev/How-to-avoid-splunk-from-indexing-duplicate-data-Sending-using/m-p/512808#M9151 LAcioffi 2020-08-06T16:38:23Z https://community.splunk.com/t5/Splunk-Dev/How-to-avoid-splunk-from-indexing-duplicate-data-Sending-using/m-p/512830#M9152 Splunk does not and can not filter duplicates on ingest. Thu, 06 Aug 2020 18:06:34 GMT https://community.splunk.com/t5/Splunk-Dev/How-to-avoid-splunk-from-indexing-duplicate-data-Sending-using/m-p/512830#M9152 richgalloway 2020-08-06T18:06:34Z