Building for the Splunk Platform

How to avoid splunk from indexing duplicate data? (Sending using python + HTML)

LAcioffi
Explorer

Hello everyone!

I am using a python script to send some data to a splunk instance on my own machine. The code is something like this:

service = client.connect(host='localhost',port=8089,username='SOMETHING',password='SOMETHING2')
myindex = service.indexes["indexName"]
myindex.submit(jsonData, sourcetype="bobby", host="local")

 

Is there any way to "tell" Splunk not to index anything duplicated? That is, anything that already exists in the index.

I know i could mess with the script to avoid sending duplicates, but if splunk could do it, things would be easier.

Thanks! 

Labels (1)
Tags (3)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust
Splunk does not and can not filter duplicates on ingest.
---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust
Splunk does not and can not filter duplicates on ingest.
---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...