Getting Data In

Is it possible to dedup events before they are indexed?

nijjie
Engager

Using

index=ets2  source="my_source" | eval id=_cd."|".index."|".splunk_server | transaction _raw maxspan=1s keepevicted=true mvlist=t | search eventcount>1 | eval delete_id=mvindex(id, 1, -1) | stats count by delete_id | fields - count

I have approx. 500,000 events in 24 hrs that are duplicates. I would like to dedup prior to indexing. Is this possible?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

I don't think Splunk can identify/remove duplicate during indexing. The options would to remove duplicate at the source which is generating the log or pre-process the log after is generated and before it's indexed.

0 Karma

hartfoml
Motivator

interesting question as to why the system is writing duplicate logs or are the time stamps different on each of the logs. this could be a case where the system is writing the same log _id every time it finds it but with different time stamps. It's not like a machine to make a mistake but rather the programmer could tell the machine to write the logs in this unusual fashion.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...