Splunk Search

How to exclude duplicates when using outputlookup ?

MeMilo09
Path Finder

Hello All, 

I have a lookup that is a saved as a schedule report that runs once a week.  This schedule report will get the new email addresses that were populated upon the search, then write the new email addresses to another lookup. The issue I have is that I get duplicates as this search runs once a week.  Is there a way I can avoid duplicates using outputlookup?  Dedup is not doing the trick...

| inputlookup Stored_Email_lookups.csv
| table Email, User_Id | rename User_Id as "New User" | dedup Email  | outputlookup  append=true "New_Incoming_Emails.csv"
Labels (1)
Tags (2)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @MeMilo09,

you have to filter the result for lookup updating using the lookup itself.

So, if you take the informations Email User_Id from thevents of an index, you could run something like this:

index=your_index NOT [ | inputlookup Stored_Email_lookups.csv | fields Email User_Id ]
| dedup Email User_Id 
| table Email User_Id 
| outputlookup  append=true Stored_Email_lookups.csv

Ciao.

Giuseppe

View solution in original post

johnhuang
Motivator

This is how I would normally do it. It avoids doing a subsearch.

index=your_index
| append [| inputlookup Stored_Email_lookups.csv]
| dedup Email User_Id 
| table Email User_Id 
| outputlookup Stored_Email_lookups.csv
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @johnhuang,

you are using a sub search as mine!

anyway, it's another similar solution: you rebuild every time the full lookup.

Ciao.

Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @MeMilo09,

you have to filter the result for lookup updating using the lookup itself.

So, if you take the informations Email User_Id from thevents of an index, you could run something like this:

index=your_index NOT [ | inputlookup Stored_Email_lookups.csv | fields Email User_Id ]
| dedup Email User_Id 
| table Email User_Id 
| outputlookup  append=true Stored_Email_lookups.csv

Ciao.

Giuseppe

gcusello
SplunkTrust
SplunkTrust

Hi @MeMilo09,

good for your, see next time!

Ciao and happy splunking.

Giuseppe

P.S.:. Karma Points are appreciated 😉

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud’s AI Assistant in Action Series: Analyzing and ...

This is the second post in our Splunk Observability Cloud’s AI Assistant in Action series, in which we look at ...

Elevate Your Organization with Splunk’s Next Platform Evolution

 Thursday, July 10, 2025  |  11AM PDT / 2PM EDT Whether you're managing complex deployments or looking to ...

Splunk Answers Content Calendar, June Edition

Get ready for this week’s post dedicated to Splunk Dashboards! We're celebrating the power of community by ...