topic Re: How to Delete Duplicate Events in Getting Data In

How to Delete Duplicate Events

I-Man — Tue, 05 Apr 2011 01:46:30 GMT

We have a search that captures the count per host every 10 minutes and puts the results into a summary index. For some reason an event for a single host count during a 10 minute time period was indexed 27 times. Every event is exactly the same. So in order to correct our stats, 26 of these events need to be deleted...How do i do that? I tried:

... | head 26 | delete

This returned an error: Error in 'delete' command: This command cannot be invoked after the non-streaming command 'head'.

Again, I have 27 events that are exactly the same. How can i delete all except 1? Thanks in advance for any help.

I-Man

Re: How to Delete Duplicate Events

hazekamp — Tue, 05 Apr 2011 02:56:10 GMT

I-Man,

If these 27 events are identical and you need to delete 26 of them I would recommend this approach:

1.  Determine the "event_id" of the event you would like to keep.  To do this, perform your search with the following eval statement to create a unique identifier:
<my 27 events> | eval event_id=splunk_server."@@".index."@@"._cd

2.  Delete the other 26 events "NOT event_id=<your_event_id>":
<my 27 events> | eval event_id=splunk_server."@@".index."@@"._cd | search NOT event_id=macfish@@_internal@@2:1309157 | delete

CAUTION: Make sure the first part of your search identifies only the 27 events you are interested in deleting....or the "search NOT event_id=" will delete more than expected.

Re: How to Delete Duplicate Events

I-Man — Tue, 05 Apr 2011 03:22:49 GMT

Perfect...YTMND

Re: How to Delete Duplicate Events

ephemeric — Wed, 24 Oct 2012 15:07:13 GMT

Would it be possible to run this search across the board and delete the output from:

soucetype="ds*" | stats count values(host) values(source) values(sourcetype) by _raw | where count > 1

Also, could one run a search and output that into a table and then proceed to do deletions?

Thank you, we are stuck this side!

Re: How to Delete Duplicate Events

roryab — Fri, 30 Nov 2012 17:34:18 GMT

To remove duplicate events with different timestamps it is possible using a subsearch and dedup:

sourcetype=fs_notification NOT [search sourcetype=fs_notification | dedup path action modtime sortby +_time] | delete

This will remove duplicated fs change events that have different timestamps, the same path, action and modtime whilst keeping the oldest.

Re: How to Delete Duplicate Events

joao_amorim — Fri, 10 Jul 2015 10:24:23 GMT

If you want the last why not just do | tail 1 ?
And with that you have the last event

Re: How to Delete Duplicate Events

vinkumar_splunk — Fri, 13 Jul 2018 05:08:44 GMT

Error in 'delete' command: This command cannot be invoked after the non-streaming command 'tail'.