topic Re: Dedup duplicates in Splunk Search

Dedup duplicates

HeinzWaescher — Wed, 12 Mar 2014 13:50:02 GMT

Hi,

what is the easiest way to filter out event duplicates without adding every field in the dedup command?
Is
| dedup _raw

the correct approach?

Re: Dedup duplicates

Ayn — Wed, 12 Mar 2014 13:56:08 GMT

dedup _raw should work just fine, yes.

Re: Dedup duplicates

HeinzWaescher — Wed, 12 Mar 2014 14:18:39 GMT

great, thanks

Re: Dedup duplicates

HeinzWaescher — Thu, 13 Mar 2014 11:07:16 GMT

I've got two additional questions regarding this topic:

How can I search for the count of events that have duplicates?
How can I search for the total number of duplicates?

Heinz

Re: Dedup duplicates

Rocket66 — Thu, 13 Mar 2014 12:28:22 GMT

You can count duplicated event by using the "transaction" command. And then count the events by using "eventcount"

eg.:

eventtype="*" | transaction session_id | Where eventcount>1 | stats count by eventcount

to find out how many duplicates occured

or:

eventtype="*" | transaction session_id | Where eventcount>1 | stats count(eventcount)

to count how many different duplicated events occured

or ...

Re: Dedup duplicates

HeinzWaescher — Fri, 14 Mar 2014 13:31:00 GMT

Unfortunately I don't have an unique identifier for each event like your proposed session_id

Re: Dedup duplicates

ITUser1 — Mon, 27 Apr 2015 15:00:29 GMT

When I try and enter the "|dedup _raw" command at the end of my search parameter I end up with no matches but when I take it off the end I end up with thousands. I can see that they are duplicates(same IP address, name, and port) but it still doesn't work. any suggestions?