Getting Data In

How to append in a csv file only records which are unique from a certain point of time?

skender27
Contributor

Hi,

I need to append in a csv file only records which are unique from a certain date/time.
The aim is to have only new events added to the csv file (and so the search would be scheduled)
I used the outputlookup append=true MyFile.csv, but that appends results every time, including the previous one.
Is there a way to put in the outputlookup comand criteria about other fields (such as _time or created_date...)?

The only way I am thinking is fixing the timerange of the search which charges the csv file...

Any suggestions?

Thanks,
Skender

0 Karma

woodcock
Esteemed Legend

If I understand you correctly, you can just do something like this;

| inputcsv max=0 MyFile.csv | append [ <your search here but must return no more than 50K events> ] | sort _time | dedup <list of your fields whose most recent combination of values you need to track> | outputcsv MyFile.csv
0 Karma

woodcock
Esteemed Legend

Why not just read it in, then append on to it, then dedup it and finally write it all back out?

0 Karma
Get Updates on the Splunk Community!

Security Highlights: September 2022 Newsletter

 September 2022 The Splunk App for Fraud Analytics (SFA) is now Splunk SupportedUse your existing Splunk ...

Platform Highlights | September 2022 Newsletter

 September 2022 What’s New in 9.0 and How to UpgradeGet a walk through of what is new Splunk Enterprise 9.0 ...

Observability Highlights | September 2022 Newsletter

 September 2022 Splunk Observability SuiteAccess to "Classic" SignalFx Interface Will be Removed on Sept 30, ...