We have high cardinality data -- virtually every event is unique except for a small percentage of cases that we care about. So we're finding that we have count the unique ids, track them somehow in order to find the duplicates. Its just not feasible in Splunk when we have millions of events per minute.
Example:
search | stats count by unique_id | where count>1
(Millions of events per minute, results in a few hundred events where count>1). Summary indexing is not really a solution here since the unique_id could cross time/minute boundary.
