How to avoid splunk from indexing duplicate data? (Sending using python + HTML)

[LAcioffi]

Hello everyone!

I am using a python script to send some data to a splunk instance on my own machine. The code is something like this:

service = client.connect(host='localhost',port=8089,username='SOMETHING',password='SOMETHING2')
myindex = service.indexes["indexName"]
myindex.submit(jsonData, sourcetype="bobby", host="local")

Is there any way to "tell" Splunk not to index anything duplicated? That is, anything that already exists in the index.

I know i could mess with the script to avoid sending duplicates, but if splunk could do it, things would be easier.

Thanks! 

[richgalloway · Accepted solution]

Splunk does not and can not filter duplicates on ingest.