All Apps and Add-ons

Avoid duplicate entries of json data into splunk

hashsplunk
Loves-to-Learn Lots

I have a python script to call the api everyday . Whenever the script runs , it calls the api and fetches all time data . The issue i have here is to avoid duplicate data when it gets ingested into splunk.I have a unique field in the json data to avoid the duplicate entries .How do i configure that in props.conf ? Can you please help

Labels (1)
0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @hashsplunk,

It is not possible to check duplicate new events with already indexed events. You can do this in your script, keep the latest unique field in a file and send only events after this event to Splunk.

If this reply helps you an upvote and "Accept as Solution" is appreciated.
0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

Observe and Secure All Apps with Splunk

 Join Us for Our Next Tech Talk: Observe and Secure All Apps with SplunkAs organizations continue to innovate ...

What's New in Splunk Observability - August 2025

What's New We are excited to announce the latest enhancements to Splunk Observability Cloud as well as what is ...

Introduction to Splunk AI

How are you using AI in Splunk? Whether you see AI as a threat or opportunity, AI is here to stay. Lucky for ...