we have two indexes with some overlap in fields. specifically IP addresses. what I would like to is do an initial search dedup all the dest_ips then in the same search string use the deduped list of IPs as the filter on the second index. Is this possible? I would think some type of lookup table is the way to go.
It depends on the size of the result set and the frequency with which you'll run this. If you want to save the results of that first deduped query to use over and over, the lookup file would be a good way to go:
index=first_index | stats values(dest_ip) AS dest_ip | outputlookup ip_list
And then to use it in your subsequent searches:
index=second_index [ | inputlookup ip_list | stats values(dest_ip) AS dest_ip | format ]
If it's just an ad-hoc thing you want to run once, you are likely fine with a single subsearch:
index=second_index [ search index=first_index | stats values(dest_ip) AS dest_ip | format ]