https://community.splunk.com/t5/Splunk-Search/How-to-delete-duplicates-from-Lookup-csv-file/m-p/584907#M203756 <P>Change the saved search or post-process the saved search to remove duplicates before writing the csv.</P><P>There a number of ways to remove duplicates depending on your criteria. For example, when there is a "duplicate", is it completely duplicated across all fields or a subset? If it is a subset, which version takes priority, e.g. first, last, max, min, etc.? If it is not a subset, is the order in anyway significant (unlikely if being used as a lookup but worth considering anyway)?</P> Mon, 14 Feb 2022 06:59:24 GMT ITWhisperer 2022-02-14T06:59:24Z https://community.splunk.com/t5/Splunk-Search/How-to-delete-duplicates-from-Lookup-csv-file/m-p/584901#M203755 <P>Hello,&nbsp; We have a CSV Lookup file that is getting populated by a saved search.&nbsp; We are noticing there are lot of duplicate rows getting created every other day.&nbsp; &nbsp;The file doesn't open in Lookup Editor App as its size is &gt;&nbsp; 10MB.&nbsp; &nbsp; Can someone pls advise how to delete duplicates via a query ?<BR /><BR /><BR /></P> Mon, 14 Feb 2022 06:29:59 GMT https://community.splunk.com/t5/Splunk-Search/How-to-delete-duplicates-from-Lookup-csv-file/m-p/584901#M203755 neerajs_81 2022-02-14T06:29:59Z https://community.splunk.com/t5/Splunk-Search/How-to-delete-duplicates-from-Lookup-csv-file/m-p/584907#M203756 <P>Change the saved search or post-process the saved search to remove duplicates before writing the csv.</P><P>There a number of ways to remove duplicates depending on your criteria. For example, when there is a "duplicate", is it completely duplicated across all fields or a subset? If it is a subset, which version takes priority, e.g. first, last, max, min, etc.? If it is not a subset, is the order in anyway significant (unlikely if being used as a lookup but worth considering anyway)?</P> Mon, 14 Feb 2022 06:59:24 GMT https://community.splunk.com/t5/Splunk-Search/How-to-delete-duplicates-from-Lookup-csv-file/m-p/584907#M203756 ITWhisperer 2022-02-14T06:59:24Z https://community.splunk.com/t5/Splunk-Search/How-to-delete-duplicates-from-Lookup-csv-file/m-p/584914#M203759 <P>I have actually updated the problem scenario in another post and tagged you in it.&nbsp; Just' realized its not really duplicates but results getting appended to data in previous row. Pls see below. Can you help ?<BR /><BR /><A href="https://community.splunk.com/t5/Splunk-Search/How-to-make-a-Search-NOT-append-results-from-previous-row-if/m-p/584912#M203757" target="_blank">https://community.splunk.com/t5/Splunk-Search/How-to-make-a-Search-NOT-append-results-from-previous-row-if/m-p/584912#M203757</A></P> Mon, 14 Feb 2022 07:21:16 GMT https://community.splunk.com/t5/Splunk-Search/How-to-delete-duplicates-from-Lookup-csv-file/m-p/584914#M203759 neerajs_81 2022-02-14T07:21:16Z