https://community.splunk.com/t5/Splunk-Search/How-does-dedup-work-in-splunk/m-p/161824#M45739 <P>It can be expensive, yes, as it needs to save the every unique entry in a temporary place to keep comparing with every following event. To see how expensive it is, just use the Job inspector, it'll show how long each command takes to run.</P> <P>Also, remember that deleting the record, doesn't actually delete anything, just mark it so won't show up again... but still very handy in your situation as you won't need to re-run dedup every time.</P> <P>Cheers</P> Mon, 02 Mar 2015 23:06:19 GMT musskopf 2015-03-02T23:06:19Z https://community.splunk.com/t5/Splunk-Search/How-does-dedup-work-in-splunk/m-p/161823#M45738 <P>How does dedup work in splunk ? My concern is about the performance.<BR /> If my search is over 500K -1M events out of which 2K events are duplicates, is using dedup going to be expensive ? Or should I find a way way to delete those 2K events and avoid using dedup ?</P> <P>Can someone give me suggestions on this or direct me to a discussion where I can find the answer to this question.</P> Mon, 02 Mar 2015 17:14:56 GMT https://community.splunk.com/t5/Splunk-Search/How-does-dedup-work-in-splunk/m-p/161823#M45738 nibinabr 2015-03-02T17:14:56Z https://community.splunk.com/t5/Splunk-Search/How-does-dedup-work-in-splunk/m-p/161824#M45739 <P>It can be expensive, yes, as it needs to save the every unique entry in a temporary place to keep comparing with every following event. To see how expensive it is, just use the Job inspector, it'll show how long each command takes to run.</P> <P>Also, remember that deleting the record, doesn't actually delete anything, just mark it so won't show up again... but still very handy in your situation as you won't need to re-run dedup every time.</P> <P>Cheers</P> Mon, 02 Mar 2015 23:06:19 GMT https://community.splunk.com/t5/Splunk-Search/How-does-dedup-work-in-splunk/m-p/161824#M45739 musskopf 2015-03-02T23:06:19Z