topic Re: How to remove duplicate event data from index? in Splunk Dev

How to remove duplicate event data from index?

zpavic — Thu, 21 Apr 2011 19:32:24 GMT

First of all, I need ask a question because I don't have enough karma points for upload an app.

I had some little problems with input script in Splunk which created duplicate event data logs in indexes. This is a reason because I wrote python script to remove that duplicate events.

Script works for me! Enjoy! 😉

Re: How to remove duplicate event data from index?

John_Mark — Thu, 21 Apr 2011 20:11:39 GMT

Wait - the system isn't supposed to prevent you from uploading an app. That's a bug 😞

Let me get on that.

Re: How to remove duplicate event data from index?

zpavic — Thu, 21 Apr 2011 21:26:26 GMT

I logout - login again and then I could upload an app but when I uploaded my app disappeared 😞

I will try tomorrow!

Re: How to remove duplicate event data from index?

John_Mark — Thu, 21 Apr 2011 21:29:15 GMT

Hi - I saw the app you uploaded and approved it. It will appear in a few minutes.

Sounds like we need to improve the process for uploading apps 🙂

Re: How to remove duplicate event data from index?

zpavic — Fri, 22 Apr 2011 04:18:07 GMT

Thanks for approving! 🙂

Re: How to remove duplicate event data from index?

yannK — Wed, 12 Oct 2011 23:58:06 GMT

Worked well, thanks.
It can take a long time to run if your data is large, some time restrictions may be useful.

To generalize to any duplicate (not only in a short time range) I modified the search1 line 59

search1 = 'search index=' + index + ' | transaction _raw keepevicted=true | where eventcount>1'

Re: How to remove duplicate event data from index?

jrodman — Thu, 08 Dec 2011 00:41:22 GMT

Unfortunately this script is only correct if you only have a maximum of one valid event per second in the targetted index. In other words, any other events in that second are removed by the script, even if they are not duplicates.

Re: How to remove duplicate event data from index?

Dark_Ichigo — Thu, 05 Apr 2012 00:58:32 GMT

You could probably use the dedup command at search time: | dedup _raw

This will remove all duplicate data from your index

Dedup Command

Re: How to remove duplicate event data from index?

richprescott — Wed, 06 Jun 2012 23:17:27 GMT

Using dedup on just the _raw data could remove events that are not duplicates. You would need to include the other indexed fields to ensure uniqueness. For example, if you have an error message being logged multiple times and then used dedup just on the _raw data, you would only see one occurrence of that error in Splunk. Including _time would help, but only for that server, so you would also need to include host. Then you run into issues if there are two of the same errors on the same host at the same time, etc. ad infinitum.

Re: How to remove duplicate event data from index?

Dark_Ichigo — Thu, 07 Jun 2012 00:10:39 GMT

Yep, already been through that, I did realize however that splunk was indexing some files more than once which was the main cause of the problem.

I stopped using dedup after that.

Thanks 🙂

Re: How to remove duplicate event data from index?

jmorais — Mon, 18 Sep 2017 22:28:37 GMT

I downvoted this post because dedup dont remove from index, remove from search...