Getting Data In

how to delete duplicate events

gooza
Communicator

I messed up my production server - deleted by mistake some of the internal splunk indexes instead of deleting them on my TEST server. As a result of that SPLUNK re-indexed logfiles in monitored directories , same files that were already indexed in the past .

Now I have events that were indexed twice : same _raw with different _indextime - one _indextime from before I've deleted the internal indexes and one _indextime with a timestamp after the internal indexes deletion.

How can I correct the mistake and remove duplicate events from my production indexes ?

what is the best query I can run to detect only the events with duplicate _indextime and delete only one of the duplicates - the one with the later _indextime

Michael
Contributor

three steps: 1) By default, you cannot delete events. You have to edit your role (Manager, Access Controls, Users, select the account, assign the role of can_delete) -- Splunk recommends creating a separate user account for deleting events, however, I just assign it to the account I use every day... 2) Create a query that will display the things you wanted to remove. 3) Pipe to the keyword "delete".

Example: host="tester" oops-stuff | delete

If there's a lot, it may take a while.

Some people keep saying "Splunk doesn't really delete them..." but I've seen the disk space used by events in my index go down after doing this...if it's not deleting them, then it's a pretty good imitation of it.

gooza
Communicator

my problem is not running the delete command but how to "Create a query that will display the" duplicated events . how can I recognize the duplicated events so I can choose only one of them and delete that. mu thoughts are that the _raw field is the same but the _indextime field is diffrent... I can see the events but I don't know how to create the query the will give me only the events I want to delete...

0 Karma

kevintelford
Path Finder

I don't believe the data is deleted from disk, just from what is returned in a given search. The data will be deleted once the buckets containing the records are deleted, if at all by a frozen policy.

0 Karma
Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...