How to append on existing gz lookup file without temporary file

[shanmu]

Hi Team,

I have many(10+) large csv lookup files (200MB) being referred in multiple places. To improve optimisation, moved them into compressed files (*.csv.gz) and made changes in existing files.

In some cases, we need to retain the existing file data. Thus said,

| outputlookup *.csv.gz  ```will fail to retain old data```

Hence Planning to do as follows,

| inputlookup "old_file.csv.gz" | outputlookup "temp_file.csv" ```taking a backup ```
| Run a new search 
| outputlookup append=t "temp_file.csv" ```append  new search results to the temp file```
| inputlookup "temp_file.csv" | outputlookup "old_file.csv.gz"
| makeresults | where false() | outputlookup create_empty=true "temp_file.csv"

 
And this needs to be done on multiple places :(.

Is there any better way to perform this without creating/clearing temp files?

[PickleRick]

You cannot append to a gzipped file. As simple as that.