How to eliminate duplicate rows in a scheduled lookup?

[joydeep741]

I have created a search to populate a lookup periodically.

 index x sourcetype=y | outputlookup abc.csv append=true

Lookup is like

EventId, Start, End
000,1,2
111,3,5

I do not want duplicate rows for EventId. My current logic is not taking care of that.
What can I add to the search so that every time a new row gets added, Splunk should only update the existing and not add a new one if event id already exists

[woodcock · Accepted solution]

Like this:

index=x sourcetype=y
| appendpipe [| inputlookup abc.csv ]
| dedup EventId
| outputlookup abc.csv

You might also include _time and add before the outputlookup:

| where _time <= relative_time(now(), "-30d")

[somesoni2]

Give this a try

index x sourcetype=y | inputlookup abc.csv append=true | dedup EventId | outputlookup abc.csv