Splunk Search

How to exclude data that is already duplicated?

Hong_TP
Engager

Hi ,

I have somthing data need to deduplicate.

I got some data from two database and save in different indexes . I use the following SPL to merge the data as

 

 

 

index="data1"  sourcetype="data1" | append [search index="data2" sourcetype="data2"]
|rename data1DATA as 1data
|eval dataall=coalesce(1data,2data)
|table dataall sourcetype

 

 

 

and I got results like this

 

 

 

dataall      sourcetype
------       ----------
abc,1        data1
abc,1        data2
def,2        data1
abc,3        data2

 

 

 

Now, I need to compare the data and exclude duplicate data . The result is like the following

 

 

 

dataall      sourcetype
------       ----------
def,2        data1
dbc,3        data2

 

 

 

Any suggestions ?

Greetings and thanks!

Labels (1)
Tags (3)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust
| eventstats count by dataall
| where count == 1

View solution in original post

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
| eventstats count by dataall
| where count == 1
0 Karma
Get Updates on the Splunk Community!

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...