Is there a way to deduplicate event in summary index

[Daniel_Pham]

I created a report for finding list intersection of two set

A: inputlookup spam_ip (Indicator of compromise)

B: index=main (event log)

| inputlookup spam_ip | join srcip [ search index=main | rename ip  as srcip | fields srcip ] | summaryindex spool=t uselb=t addtime=t index="threat_summary" file="RMD55f183b338b214f84_487362985.stash_new" name="matches test" marker=""

and Time range : All day (because event in two sets grow daily,)

after the report  runs, it adds result into summary index. The problem is the result contains all event added before

[aasabatini]

Hi @Daniel_Pham 

why don't use a lookup command direclty?

I mean if you use static data from lookup on your search without a timestamp, is normal if you have duplicate.

Now I don't know your set of data, but try this search and let me know if works for you.

index=main | rename ip  as srcip | lookup spam_ip srcip

If this search works your schedule avoid to deduplicate the data

Regards

Alessandro

[Daniel_Pham]

Your search works, but my expected results are events from lookup not main index

This is my lookup data

asn,classification,confidence,country,date_first,date_last,detail,id,itype,lat,lon,maltype,org,resource_uri,severity,source,actor,tipreport,type,srcip,domain,md5,email,url	

And all event data contains a ip field and not the same name. 

The schedule must be All day,  because any incoming event log from main index can be in lookup,  and vice versa