How to exclude data that is already duplicated?

[Hong_TP]

Hi ,

I have somthing data need to deduplicate.

I got some data from two database and save in different indexes . I use the following SPL to merge the data as

index="data1"  sourcetype="data1" | append [search index="data2" sourcetype="data2"]
|rename data1DATA as 1data
|eval dataall=coalesce(1data,2data)
|table dataall sourcetype

and I got results like this

dataall      sourcetype
------       ----------
abc,1        data1
abc,1        data2
def,2        data1
abc,3        data2

Now, I need to compare the data and exclude duplicate data . The result is like the following

dataall      sourcetype
------       ----------
def,2        data1
dbc,3        data2

Any suggestions ?

Greetings and thanks!

[ITWhisperer · Accepted solution]

| eventstats count by dataall
| where count == 1