Thanks for the response. So the purpose is just to have a place where the observables are backed up in this index ready to be used if needed. Would also like to build dashboarding based on that index which is a little tricky based on the lookup file itself for capturing metrics, ingestion rate of IoC's etc. The ip_intel file is growing on a daily basis, being fed by various Threat Intel Sources. So When I say "new" this would be new observables/new data ingesting into that file. Ive got a little further with this but getting some errors. | inputlookup ip_intel | dedup ip | stats count by ip threat_key description | fields - count | search NOT [ index=backup_ti source=daily_ip_intel | stats count by ip threat_key description | fields - count] | collect index=backup_ti source=daily_ip_intel So trying to compare the lookup file with whats already in the index. Identify data that is not present in the backup_ti index but is actually there in the lookup file, then pulls that data down into the index.
... View more