Hi Team,
I have many(10+) large csv lookup files (200MB) being referred in multiple places. To improve optimisation, moved them into compressed files (*.csv.gz) and made changes in existing files.
In some cases, we need to retain the existing file data. Thus said,
| outputlookup *.csv.gz ```will fail to retain old data```
Hence Planning to do as follows,
| inputlookup "old_file.csv.gz" | outputlookup "temp_file.csv" ```taking a backup ```
| Run a new search
| outputlookup append=t "temp_file.csv" ```append new search results to the temp file```
| inputlookup "temp_file.csv" | outputlookup "old_file.csv.gz"
| makeresults | where false() | outputlookup create_empty=true "temp_file.csv"
And this needs to be done on multiple places :(.
Is there any better way to perform this without creating/clearing temp files?
