@okretzer As @richgalloway says, 40TB will take time regardless, however, from your original search I would suggest: Replace dedup with stats, as I believe that will run faster than dedup. stats will throw away the fields you don't need, whereas dedup has to carry all the other fields in the event, which you are discarding anyway with the table command. Also, table command is not good for large data sets, as it runs on the search head. Use fields command instead, as that runs on the indexer. Currently you are passing all the data from the indexers back to the search head before discarding it. index=myindex src_ip=*
| stats count by src_ip, dest_ip
| fields - count
| lookup mylookup.csv dest_ip OUTPUT dest_ip
| sort src_ip Do some timings and look at job inspector to get an idea.
... View more