Hi,
I need to append in a csv file only records which are unique from a certain date/time.
The aim is to have only new events added to the csv file (and so the search would be scheduled)
I used the outputlookup append=true MyFile.csv
, but that appends results every time, including the previous one.
Is there a way to put in the outputlookup comand criteria about other fields (such as _time or created_date...)?
The only way I am thinking is fixing the timerange of the search which charges the csv file...
Any suggestions?
Thanks,
Skender
If I understand you correctly, you can just do something like this;
| inputcsv max=0 MyFile.csv | append [ <your search here but must return no more than 50K events> ] | sort _time | dedup <list of your fields whose most recent combination of values you need to track> | outputcsv MyFile.csv
Why not just read it in, then append on to it, then dedup it and finally write it all back out?