Splunk Search

How to delete duplicates from Lookup csv file ?

neerajs_81
Builder

Hello,  We have a CSV Lookup file that is getting populated by a saved search.  We are noticing there are lot of duplicate rows getting created every other day.   The file doesn't open in Lookup Editor App as its size is >  10MB.    Can someone pls advise how to delete duplicates via a query ?


Labels (1)

ITWhisperer
SplunkTrust
SplunkTrust

Change the saved search or post-process the saved search to remove duplicates before writing the csv.

There a number of ways to remove duplicates depending on your criteria. For example, when there is a "duplicate", is it completely duplicated across all fields or a subset? If it is a subset, which version takes priority, e.g. first, last, max, min, etc.? If it is not a subset, is the order in anyway significant (unlikely if being used as a lookup but worth considering anyway)?

neerajs_81
Builder

I have actually updated the problem scenario in another post and tagged you in it.  Just' realized its not really duplicates but results getting appended to data in previous row. Pls see below. Can you help ?

https://community.splunk.com/t5/Splunk-Search/How-to-make-a-Search-NOT-append-results-from-previous-...

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...

Splunk AppDynamics Agents Webinar Series

Mark your calendars! On June 24th at 12PM PST, we’re going live with the second session of our Splunk ...

SplunkTrust Application Period is Officially OPEN!

It's that time, folks! The application/nomination period for the 2025 SplunkTrust is officially open! If you ...