To find these events, you can run the following search
...|eventstats count as duplicate by _raw _host _time | where duplicate>1
As a temporary measure you can remove the duplicates from each search with the dedup command
...| dedup _raw _host _time
BUT this is inefficient so you need to prevent and get rid of the duplicates. If you have multiple indexers, look for data going to more than one, look for almost duplicate files, avoid using crcsalt in inputs.conf etc.
Once you have got rid of the cause, get rid of the duplicates using the following search
* | eventstats count as duplicates first(_cd) as cd by _raw host _time | where cd!=_cd
I have deliberately not joined the delete to the above search as it is good practice to check the data before deleting it. Confirm it is only bringing back duplicates and not the original then pipe to delete. You will need to temporarily add the candelete roll to your account for this to work.
...| delete
... View more