How to Delete Duplicate Events

[I-Man]

We have a search that captures the count per host every 10 minutes and puts the results into a summary index. For some reason an event for a single host count during a 10 minute time period was indexed 27 times. Every event is exactly the same. So in order to correct our stats, 26 of these events need to be deleted...How do i do that? I tried:

... | head 26 | delete

This returned an error: Error in 'delete' command: This command cannot be invoked after the non-streaming command 'head'.

Again, I have 27 events that are exactly the same. How can i delete all except 1? Thanks in advance for any help.

I-Man

[joao_amorim]

If you want the last why not just do | tail 1 ?
And with that you have the last event

[vinkumar_splunk]

Error in 'delete' command: This command cannot be invoked after the non-streaming command 'tail'.

[roryab]

To remove duplicate events with different timestamps it is possible using a subsearch and dedup:

sourcetype=fs_notification NOT [search sourcetype=fs_notification | dedup path action modtime sortby +_time] | delete

This will remove duplicated fs change events that have different timestamps, the same path, action and modtime whilst keeping the oldest.

[hazekamp · Accepted solution]

I-Man,

If these 27 events are identical and you need to delete 26 of them I would recommend this approach:

1.  Determine the "event_id" of the event you would like to keep.  To do this, perform your search with the following eval statement to create a unique identifier:
<my 27 events> | eval event_id=splunk_server."@@".index."@@"._cd

2.  Delete the other 26 events "NOT event_id=<your_event_id>":
<my 27 events> | eval event_id=splunk_server."@@".index."@@"._cd | search NOT event_id=macfish@@_internal@@2:1309157 | delete

CAUTION: Make sure the first part of your search identifies only the 27 events you are interested in deleting....or the "search NOT event_id=" will delete more than expected.

[ephemeric]

Would it be possible to run this search across the board and delete the output from:

soucetype="ds*" | stats count values(host) values(source) values(sourcetype) by _raw | where count > 1

Also, could one run a search and output that into a table and then proceed to do deletions?

Thank you, we are stuck this side!

[I-Man]

Perfect...YTMND