Getting Data In

How to append in a csv file only records which are unique from a certain point of time?

skender27
Contributor

Hi,

I need to append in a csv file only records which are unique from a certain date/time.
The aim is to have only new events added to the csv file (and so the search would be scheduled)
I used the outputlookup append=true MyFile.csv, but that appends results every time, including the previous one.
Is there a way to put in the outputlookup comand criteria about other fields (such as _time or created_date...)?

The only way I am thinking is fixing the timerange of the search which charges the csv file...

Any suggestions?

Thanks,
Skender

0 Karma

woodcock
Esteemed Legend

If I understand you correctly, you can just do something like this;

| inputcsv max=0 MyFile.csv | append [ <your search here but must return no more than 50K events> ] | sort _time | dedup <list of your fields whose most recent combination of values you need to track> | outputcsv MyFile.csv
0 Karma

woodcock
Esteemed Legend

Why not just read it in, then append on to it, then dedup it and finally write it all back out?

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In January, the Splunk Threat Research Team had one release of new security content via the Splunk ES Content ...

Expert Tips from Splunk Professional Services, Ensuring Compliance, and More New ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Observability Release Update: AI Assistant, AppD + Observability Cloud Integrations & ...

This month’s releases across the Splunk Observability portfolio deliver earlier detection and faster ...