Hi @sumarri, I created a dummy search to mock up your data, and created a lookup with 104,000 entries: | makeresults count=140000
| streamstats count as id
| eval account="account" . substr("000...
See more...
Hi @sumarri, I created a dummy search to mock up your data, and created a lookup with 104,000 entries: | makeresults count=140000
| streamstats count as id
| eval account="account" . substr("000000000".tostring(id),-6), keep="true"
| table account, keep
| outputlookup "accounts_to_keep.csv" This will be our lookup file, replicating what you have in your lookup. It has the account ID and a "keep" field, and that's it. Next, I created a dummy search to generate a bunch of data, with accounts we don't care about and the 104,000 we do care about: | makeresults count=200000
| streamstats count as id
| eval account="account" . substr("000000000".tostring(id),-6)
| eval data=random()%10000, label="whatever", _time=relative_time(now(), "-" + tostring(random()%1000) + "m")
| table account, data, label, _time To use the lookup to identify the accounts we want to keep you can use this SPL: | inputlookup accounts_to_keep.csv append=t
``` use eventstats if stats messes up your data
| eventstats values(keep) as keep by account
```
| stats values(*) as * by account
| search keep="true"
| fields - keep This add the contents of the lookup to the results (append=t) Then we use stats to combine the keep field with the events in the search If this messes up your data, you can run eventstats instead, but that may run into memory issues with massive result sets. Finally, we search for all the events where the keep field is set to "true" Depending on how big your lookup gets, you may want to make the lookup a KV store collection.