Splunk Search

How to exclude duplicates when using outputlookup ?

MeMilo09
Path Finder

Hello All, 

I have a lookup that is a saved as a schedule report that runs once a week.  This schedule report will get the new email addresses that were populated upon the search, then write the new email addresses to another lookup. The issue I have is that I get duplicates as this search runs once a week.  Is there a way I can avoid duplicates using outputlookup?  Dedup is not doing the trick...

| inputlookup Stored_Email_lookups.csv
| table Email, User_Id | rename User_Id as "New User" | dedup Email  | outputlookup  append=true "New_Incoming_Emails.csv"
Labels (1)
Tags (2)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @MeMilo09,

you have to filter the result for lookup updating using the lookup itself.

So, if you take the informations Email User_Id from thevents of an index, you could run something like this:

index=your_index NOT [ | inputlookup Stored_Email_lookups.csv | fields Email User_Id ]
| dedup Email User_Id 
| table Email User_Id 
| outputlookup  append=true Stored_Email_lookups.csv

Ciao.

Giuseppe

View solution in original post

johnhuang
Motivator

This is how I would normally do it. It avoids doing a subsearch.

index=your_index
| append [| inputlookup Stored_Email_lookups.csv]
| dedup Email User_Id 
| table Email User_Id 
| outputlookup Stored_Email_lookups.csv
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @johnhuang,

you are using a sub search as mine!

anyway, it's another similar solution: you rebuild every time the full lookup.

Ciao.

Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @MeMilo09,

you have to filter the result for lookup updating using the lookup itself.

So, if you take the informations Email User_Id from thevents of an index, you could run something like this:

index=your_index NOT [ | inputlookup Stored_Email_lookups.csv | fields Email User_Id ]
| dedup Email User_Id 
| table Email User_Id 
| outputlookup  append=true Stored_Email_lookups.csv

Ciao.

Giuseppe

gcusello
SplunkTrust
SplunkTrust

Hi @MeMilo09,

good for your, see next time!

Ciao and happy splunking.

Giuseppe

P.S.:. Karma Points are appreciated 😉

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...