Getting Data In

How to append in a csv file only records which are unique from a certain point of time?

skender27
Contributor

Hi,

I need to append in a csv file only records which are unique from a certain date/time.
The aim is to have only new events added to the csv file (and so the search would be scheduled)
I used the outputlookup append=true MyFile.csv, but that appends results every time, including the previous one.
Is there a way to put in the outputlookup comand criteria about other fields (such as _time or created_date...)?

The only way I am thinking is fixing the timerange of the search which charges the csv file...

Any suggestions?

Thanks,
Skender

0 Karma

woodcock
Esteemed Legend

If I understand you correctly, you can just do something like this;

| inputcsv max=0 MyFile.csv | append [ <your search here but must return no more than 50K events> ] | sort _time | dedup <list of your fields whose most recent combination of values you need to track> | outputcsv MyFile.csv
0 Karma

woodcock
Esteemed Legend

Why not just read it in, then append on to it, then dedup it and finally write it all back out?

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...