topic Re: How to eliminate duplicate rows in a scheduled lookup? in Splunk Search

How to eliminate duplicate rows in a scheduled lookup?

joydeep741 — Mon, 16 Jul 2018 10:30:36 GMT

I have created a search to populate a lookup periodically.

 index x sourcetype=y | outputlookup abc.csv append=true

Lookup is like

EventId, Start, End
000,1,2
111,3,5

I do not want duplicate rows for EventId. My current logic is not taking care of that.
What can I add to the search so that every time a new row gets added, Splunk should only update the existing and not add a new one if event id already exists

Re: How to eliminate duplicate rows in a scheduled lookup?

somesoni2 — Mon, 16 Jul 2018 15:57:45 GMT

Give this a try

index x sourcetype=y | inputlookup abc.csv append=true | dedup EventId | outputlookup abc.csv

Re: How to eliminate duplicate rows in a scheduled lookup?

woodcock — Mon, 16 Jul 2018 21:47:20 GMT

Like this:

index=x sourcetype=y
| appendpipe [| inputlookup abc.csv ]
| dedup EventId
| outputlookup abc.csv

You might also include _time and add before the outputlookup:

| where _time <= relative_time(now(), "-30d")