Splunk Search

How to exclude duplicates when using outputlookup ?

MeMilo09
Path Finder

Hello All, 

I have a lookup that is a saved as a schedule report that runs once a week.  This schedule report will get the new email addresses that were populated upon the search, then write the new email addresses to another lookup. The issue I have is that I get duplicates as this search runs once a week.  Is there a way I can avoid duplicates using outputlookup?  Dedup is not doing the trick...

| inputlookup Stored_Email_lookups.csv
| table Email, User_Id | rename User_Id as "New User" | dedup Email  | outputlookup  append=true "New_Incoming_Emails.csv"
Labels (1)
Tags (2)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @MeMilo09,

you have to filter the result for lookup updating using the lookup itself.

So, if you take the informations Email User_Id from thevents of an index, you could run something like this:

index=your_index NOT [ | inputlookup Stored_Email_lookups.csv | fields Email User_Id ]
| dedup Email User_Id 
| table Email User_Id 
| outputlookup  append=true Stored_Email_lookups.csv

Ciao.

Giuseppe

View solution in original post

johnhuang
Motivator

This is how I would normally do it. It avoids doing a subsearch.

index=your_index
| append [| inputlookup Stored_Email_lookups.csv]
| dedup Email User_Id 
| table Email User_Id 
| outputlookup Stored_Email_lookups.csv
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @johnhuang,

you are using a sub search as mine!

anyway, it's another similar solution: you rebuild every time the full lookup.

Ciao.

Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @MeMilo09,

you have to filter the result for lookup updating using the lookup itself.

So, if you take the informations Email User_Id from thevents of an index, you could run something like this:

index=your_index NOT [ | inputlookup Stored_Email_lookups.csv | fields Email User_Id ]
| dedup Email User_Id 
| table Email User_Id 
| outputlookup  append=true Stored_Email_lookups.csv

Ciao.

Giuseppe

gcusello
SplunkTrust
SplunkTrust

Hi @MeMilo09,

good for your, see next time!

Ciao and happy splunking.

Giuseppe

P.S.:. Karma Points are appreciated 😉

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...