https://community.splunk.com/t5/Splunk-Search/How-to-exclude-duplicates-when-using-outputlookup/m-p/584058#M203398 <P>This is how I would normally do it. It avoids doing a subsearch.</P><LI-CODE lang="markup">index=your_index | append [| inputlookup Stored_Email_lookups.csv] | dedup Email User_Id | table Email User_Id | outputlookup Stored_Email_lookups.csv</LI-CODE> Tue, 08 Feb 2022 08:16:51 GMT johnhuang 2022-02-08T08:16:51Z https://community.splunk.com/t5/Splunk-Search/How-to-exclude-duplicates-when-using-outputlookup/m-p/584032#M203385 <P>Hello All,&nbsp;</P><P>I have a lookup that is a saved as a schedule report that runs once a week.&nbsp; This schedule report will get the new email addresses that were populated upon the search, then write the new email addresses to another lookup. The issue I have is that I get duplicates as this search runs once a week.&nbsp; Is there a way I can avoid duplicates using outputlookup?&nbsp; Dedup is not doing the trick...<BR /><BR /></P><LI-CODE lang="markup">| inputlookup Stored_Email_lookups.csv | table Email, User_Id | rename User_Id as "New User" | dedup Email | outputlookup append=true "New_Incoming_Emails.csv"</LI-CODE> Tue, 08 Feb 2022 01:06:35 GMT https://community.splunk.com/t5/Splunk-Search/How-to-exclude-duplicates-when-using-outputlookup/m-p/584032#M203385 MeMilo09 2022-02-08T01:06:35Z https://community.splunk.com/t5/Splunk-Search/How-to-exclude-duplicates-when-using-outputlookup/m-p/584044#M203391 <P>Hi&nbsp;<a href="https://community.splunk.com/t5/user/viewprofilepage/user-id/233122">@MeMilo09</a>,</P><P>you have to filter the result for lookup updating using the lookup itself.</P><P>So, if you take the informations Email User_Id from thevents of an index, you could run something like this:</P><LI-CODE lang="markup">index=your_index NOT [ | inputlookup Stored_Email_lookups.csv | fields Email User_Id ] | dedup Email User_Id | table Email User_Id | outputlookup append=true Stored_Email_lookups.csv</LI-CODE><P>Ciao.</P><P>Giuseppe</P> Tue, 08 Feb 2022 07:14:29 GMT https://community.splunk.com/t5/Splunk-Search/How-to-exclude-duplicates-when-using-outputlookup/m-p/584044#M203391 gcusello 2022-02-08T07:14:29Z https://community.splunk.com/t5/Splunk-Search/How-to-exclude-duplicates-when-using-outputlookup/m-p/584058#M203398 <P>This is how I would normally do it. It avoids doing a subsearch.</P><LI-CODE lang="markup">index=your_index | append [| inputlookup Stored_Email_lookups.csv] | dedup Email User_Id | table Email User_Id | outputlookup Stored_Email_lookups.csv</LI-CODE> Tue, 08 Feb 2022 08:16:51 GMT https://community.splunk.com/t5/Splunk-Search/How-to-exclude-duplicates-when-using-outputlookup/m-p/584058#M203398 johnhuang 2022-02-08T08:16:51Z https://community.splunk.com/t5/Splunk-Search/How-to-exclude-duplicates-when-using-outputlookup/m-p/584063#M203401 <P>Hi&nbsp;<a href="https://community.splunk.com/t5/user/viewprofilepage/user-id/190794">@johnhuang</a>,</P><P>you are using a sub search as mine!</P><P>anyway, it's another similar solution: you rebuild every time the full lookup.</P><P>Ciao.</P><P>Giuseppe</P> Tue, 08 Feb 2022 09:42:38 GMT https://community.splunk.com/t5/Splunk-Search/How-to-exclude-duplicates-when-using-outputlookup/m-p/584063#M203401 gcusello 2022-02-08T09:42:38Z https://community.splunk.com/t5/Splunk-Search/How-to-exclude-duplicates-when-using-outputlookup/m-p/584206#M203433 <P>Hi&nbsp;<a href="https://community.splunk.com/t5/user/viewprofilepage/user-id/233122">@MeMilo09</a>,</P><P>good for your, see next time!</P><P>Ciao and happy splunking.</P><P>Giuseppe</P><P>P.S.:. Karma Points are appreciated <span class="lia-unicode-emoji" title=":winking_face:">😉</span></P> Wed, 09 Feb 2022 07:31:51 GMT https://community.splunk.com/t5/Splunk-Search/How-to-exclude-duplicates-when-using-outputlookup/m-p/584206#M203433 gcusello 2022-02-09T07:31:51Z