https://community.splunk.com/t5/Getting-Data-In/Indexed-data-twice-Suggestions-to-remove-data-from-being/m-p/211634#M41652 <P>Lets say we have forwarded events that are exactly the same and show in Splunk as duplicates. Running a <CODE>| dedup _raw</CODE> would resolve the duplicate events at search time. Would it make sense to run <CODE>index=main | deduce _raw | delete</CODE> so that we won't have to run a dedup every single time on that time range of events?</P> Tue, 23 Feb 2016 17:09:13 GMT ben_leung 2016-02-23T17:09:13Z https://community.splunk.com/t5/Getting-Data-In/Indexed-data-twice-Suggestions-to-remove-data-from-being/m-p/211634#M41652 <P>Lets say we have forwarded events that are exactly the same and show in Splunk as duplicates. Running a <CODE>| dedup _raw</CODE> would resolve the duplicate events at search time. Would it make sense to run <CODE>index=main | deduce _raw | delete</CODE> so that we won't have to run a dedup every single time on that time range of events?</P> Tue, 23 Feb 2016 17:09:13 GMT https://community.splunk.com/t5/Getting-Data-In/Indexed-data-twice-Suggestions-to-remove-data-from-being/m-p/211634#M41652 ben_leung 2016-02-23T17:09:13Z https://community.splunk.com/t5/Getting-Data-In/Indexed-data-twice-Suggestions-to-remove-data-from-being/m-p/211635#M41653 <P>The reason I ask this is because <CODE>| delete</CODE> would remove the events returned from the prior search. I would assume it would "delete" the duplicate AND the original events. Does anyone know the behavior of this kind of scenario? </P> Tue, 23 Feb 2016 17:12:08 GMT https://community.splunk.com/t5/Getting-Data-In/Indexed-data-twice-Suggestions-to-remove-data-from-being/m-p/211635#M41653 ben_leung 2016-02-23T17:12:08Z https://community.splunk.com/t5/Getting-Data-In/Indexed-data-twice-Suggestions-to-remove-data-from-being/m-p/211636#M41654 <P>I wouldn't advise scheduling a delete. For one, delete is expensive to run. Second, possibly dangerous in that you may wind up deleting something by accident. Third, fix the reason for duplicate events instead.</P> Tue, 23 Feb 2016 17:17:18 GMT https://community.splunk.com/t5/Getting-Data-In/Indexed-data-twice-Suggestions-to-remove-data-from-being/m-p/211636#M41654 the_wolverine 2016-02-23T17:17:18Z