how to delete duplicate events

[gooza]

I messed up my production server - deleted by mistake some of the internal splunk indexes instead of deleting them on my TEST server. As a result of that SPLUNK re-indexed logfiles in monitored directories , same files that were already indexed in the past .

Now I have events that were indexed twice : same _raw with different _indextime - one _indextime from before I've deleted the internal indexes and one _indextime with a timestamp after the internal indexes deletion.

How can I correct the mistake and remove duplicate events from my production indexes ?

what is the best query I can run to detect only the events with duplicate _indextime and delete only one of the duplicates - the one with the later _indextime

[Michael]

three steps: 1) By default, you cannot delete events. You have to edit your role (Manager, Access Controls, Users, select the account, assign the role of can_delete) -- Splunk recommends creating a separate user account for deleting events, however, I just assign it to the account I use every day... 2) Create a query that will display the things you wanted to remove. 3) Pipe to the keyword "delete".

Example: host="tester" oops-stuff | delete

If there's a lot, it may take a while.

Some people keep saying "Splunk doesn't really delete them..." but I've seen the disk space used by events in my index go down after doing this...if it's not deleting them, then it's a pretty good imitation of it.

[gooza]

my problem is not running the delete command but how to "Create a query that will display the" duplicated events . how can I recognize the duplicated events so I can choose only one of them and delete that. mu thoughts are that the _raw field is the same but the _indextime field is diffrent... I can see the events but I don't know how to create the query the will give me only the events I want to delete...

[pgoyal_splunk]

https://answers.splunk.com/answers/453991/find-the-duplicate-files-from-particular-source-in.html

[kevintelford]

I don't believe the data is deleted from disk, just from what is returned in a given search. The data will be deleted once the buckets containing the records are deleted, if at all by a frozen policy.