https://community.splunk.com/t5/Splunk-Search/Dedup-lost-data-a-bug/m-p/142525#M39583 <P>The problem is that the subsearch has a limit - and I don't see that you need the subsearch at all. You also do not need the sort, Splunk returns events in reverse time order (newest first) by default.Try this</P> <PRE><CODE>eventtype=MYTYPE | dedup fieldX </CODE></PRE> <P>This will return the most recent event for each value of <CODE>fieldX</CODE>. </P> Sat, 27 Sep 2014 20:04:11 GMT lguinn2 2014-09-27T20:04:11Z https://community.splunk.com/t5/Splunk-Search/Dedup-lost-data-a-bug/m-p/142524#M39582 <P>For below search : </P> <PRE><CODE>eventtype=MYTYPE [search eventtype=MYTYPE | sort 0 _time desc | dedup fieldX | return 1000 source] </CODE></PRE> <P>Expect to return the latest source for 1 fieldX value.</P> <P>In our data, we have over 10,000,000 events for the latest source with fieldX=A, fieldX=B, fieldX=C respectively.<BR /> Thus expect the search returns over 30,000,000 results.<BR /> However, it returns the results for fieldX=A only.</P> <P>Question : is the search correctly written? If yes, is this a bug of dedup? Is there any limitation of dedup about the result size? If we use a smaller dataset, dedup works properly with the same search.</P> Fri, 26 Sep 2014 16:04:03 GMT https://community.splunk.com/t5/Splunk-Search/Dedup-lost-data-a-bug/m-p/142524#M39582 april_tao 2014-09-26T16:04:03Z https://community.splunk.com/t5/Splunk-Search/Dedup-lost-data-a-bug/m-p/142525#M39583 <P>The problem is that the subsearch has a limit - and I don't see that you need the subsearch at all. You also do not need the sort, Splunk returns events in reverse time order (newest first) by default.Try this</P> <PRE><CODE>eventtype=MYTYPE | dedup fieldX </CODE></PRE> <P>This will return the most recent event for each value of <CODE>fieldX</CODE>. </P> Sat, 27 Sep 2014 20:04:11 GMT https://community.splunk.com/t5/Splunk-Search/Dedup-lost-data-a-bug/m-p/142525#M39583 lguinn2 2014-09-27T20:04:11Z