Building for the Splunk Platform

How to avoid splunk from indexing duplicate data? (Sending using python + HTML)

LAcioffi
Explorer

Hello everyone!

I am using a python script to send some data to a splunk instance on my own machine. The code is something like this:

service = client.connect(host='localhost',port=8089,username='SOMETHING',password='SOMETHING2')
myindex = service.indexes["indexName"]
myindex.submit(jsonData, sourcetype="bobby", host="local")

 

Is there any way to "tell" Splunk not to index anything duplicated? That is, anything that already exists in the index.

I know i could mess with the script to avoid sending duplicates, but if splunk could do it, things would be easier.

Thanks! 

Labels (1)
Tags (3)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust
Splunk does not and can not filter duplicates on ingest.
---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust
Splunk does not and can not filter duplicates on ingest.
---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

BSides Splunk 2022 - The Call for Papers is now Open!

TLDR; Main Site: https://bsidessplunk.com CFP Site: https://bsidessplunk.com/cfp CFP Opens: December 15th, ...

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...