https://community.splunk.com/t5/Getting-Data-In/Finding-and-removing-all-but-one-duplicate-doc/m-p/142392#M29116 <P>All,</P> <P>I'm curious, is there an easy way to find all duplicate logs and delete all but one of them?</P> <P>Thanks!</P> Fri, 07 Feb 2014 19:39:55 GMT bruceclarke 2014-02-07T19:39:55Z https://community.splunk.com/t5/Getting-Data-In/Finding-and-removing-all-but-one-duplicate-doc/m-p/142392#M29116 <P>All,</P> <P>I'm curious, is there an easy way to find all duplicate logs and delete all but one of them?</P> <P>Thanks!</P> Fri, 07 Feb 2014 19:39:55 GMT https://community.splunk.com/t5/Getting-Data-In/Finding-and-removing-all-but-one-duplicate-doc/m-p/142392#M29116 bruceclarke 2014-02-07T19:39:55Z https://community.splunk.com/t5/Getting-Data-In/Finding-and-removing-all-but-one-duplicate-doc/m-p/142393#M29117 <P>You could do something like this:</P> <PRE><CODE>base search | streamstats count by _raw | where count &gt; 1 </CODE></PRE> <P>That should select duplicates number 2, 3, and so on. Once you've confirmed that this really is what you're looking for, you can switch to a user with the can_delete role and pipe that to <CODE>delete</CODE>.</P> Fri, 07 Feb 2014 20:05:31 GMT https://community.splunk.com/t5/Getting-Data-In/Finding-and-removing-all-but-one-duplicate-doc/m-p/142393#M29117 martin_mueller 2014-02-07T20:05:31Z