Splunk Search

How to exclude duplicates when using outputlookup ?

MeMilo09
Path Finder

Hello All, 

I have a lookup that is a saved as a schedule report that runs once a week.  This schedule report will get the new email addresses that were populated upon the search, then write the new email addresses to another lookup. The issue I have is that I get duplicates as this search runs once a week.  Is there a way I can avoid duplicates using outputlookup?  Dedup is not doing the trick...

| inputlookup Stored_Email_lookups.csv
| table Email, User_Id | rename User_Id as "New User" | dedup Email  | outputlookup  append=true "New_Incoming_Emails.csv"
Labels (1)
Tags (2)
0 Karma
1 Solution

gcusello
Legend

Hi @MeMilo09,

you have to filter the result for lookup updating using the lookup itself.

So, if you take the informations Email User_Id from thevents of an index, you could run something like this:

index=your_index NOT [ | inputlookup Stored_Email_lookups.csv | fields Email User_Id ]
| dedup Email User_Id 
| table Email User_Id 
| outputlookup  append=true Stored_Email_lookups.csv

Ciao.

Giuseppe

View solution in original post

johnhua
Builder

This is how I would normally do it. It avoids doing a subsearch.

index=your_index
| append [| inputlookup Stored_Email_lookups.csv]
| dedup Email User_Id 
| table Email User_Id 
| outputlookup Stored_Email_lookups.csv
0 Karma

gcusello
Legend

Hi @johnhua,

you are using a sub search as mine!

anyway, it's another similar solution: you rebuild every time the full lookup.

Ciao.

Giuseppe

0 Karma

gcusello
Legend

Hi @MeMilo09,

you have to filter the result for lookup updating using the lookup itself.

So, if you take the informations Email User_Id from thevents of an index, you could run something like this:

index=your_index NOT [ | inputlookup Stored_Email_lookups.csv | fields Email User_Id ]
| dedup Email User_Id 
| table Email User_Id 
| outputlookup  append=true Stored_Email_lookups.csv

Ciao.

Giuseppe

gcusello
Legend

Hi @MeMilo09,

good for your, see next time!

Ciao and happy splunking.

Giuseppe

P.S.:. Karma Points are appreciated 😉

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...