How to append in a csv file only records which are unique from a certain point of time?

[skender27]

Hi,

I need to append in a csv file only records which are unique from a certain date/time.
The aim is to have only new events added to the csv file (and so the search would be scheduled)
I used the outputlookup append=true MyFile.csv, but that appends results every time, including the previous one.
Is there a way to put in the outputlookup comand criteria about other fields (such as _time or created_date...)?

The only way I am thinking is fixing the timerange of the search which charges the csv file...

Any suggestions?

Thanks,
Skender

[woodcock]

If I understand you correctly, you can just do something like this;

| inputcsv max=0 MyFile.csv | append [ <your search here but must return no more than 50K events> ] | sort _time | dedup <list of your fields whose most recent combination of values you need to track> | outputcsv MyFile.csv

[woodcock]

Why not just read it in, then append on to it, then dedup it and finally write it all back out?