topic Re: Dedup with multiple criteria in Splunk Search

Dedup with multiple criteria

HeinzWaescher — Wed, 15 Jan 2014 09:19:18 GMT

Hi,

I want to use the dedup command with more than one criteria.

First I used | dedup A and had 100 events afterwards.
Then I used | dedup A, B and had 70 events afterwards. In my understanding I the number of events should increase, because I've specified the dedup criteria and less duplicates should be identified?! Am I completely wrong?

Best

Heinz

Re: Dedup with multiple criteria

Ayn — Wed, 15 Jan 2014 09:43:48 GMT

Does B exist in all your events? IIRC dedup will fail otherwise.

Re: Dedup with multiple criteria

HeinzWaescher — Wed, 15 Jan 2014 10:50:33 GMT

Hey Ayn,

yes normally it should exist in all events. Is there a command to find out, whether there are events without the field B and to filter them out?

Edit:

Just tried it out with | sourctype=* AND NOT B= * .
This results in a few events

Re: Dedup with multiple criteria

Ayn — Wed, 15 Jan 2014 11:35:28 GMT

Then that's your problem there. You can do ... | fillnull B | ... if you want B with an empty value in events that don't have it. That will make dedup work.

Re: Dedup with multiple criteria

HeinzWaescher — Wed, 15 Jan 2014 11:45:03 GMT

Ah, now numbers are changing in the correct direction 🙂

And when I want to ignore events where the dedup criteria don't exist, I can just use

sourcetype=* AND
A=* AND
B=* AND

| dedup A,B

Thanks a lot!

Re: Dedup with multiple criteria

HeinzWaescher — Fri, 17 Jan 2014 13:52:23 GMT

A further question regarding the dedup command:

Let's say the fields A & B can appear multiple times in an event.
For example:

Event 1:
A=1
A=2
B=3
B=4
timestamp=X

Event:2
A=1
A=2
B=3
B=4
timestamp=X

Event 3:
A=1
A=2
B=3
B=4
timestamp=Y

| dedup A,B,timestamp

does this include all field values for A & B and results in two remaining events (event 1 and event 3)?

Thanks in advance

Heinz

Re: Dedup with multiple criteria

linu1988 — Fri, 17 Jan 2014 14:29:17 GMT

Yes it gives the value till you have something distinct with the above combination.

Re: Dedup with multiple criteria

HeinzWaescher — Tue, 21 Jan 2014 12:57:41 GMT

thanks for confirming!

Re: Dedup with multiple criteria

landen99 — Wed, 29 Apr 2015 02:37:29 GMT

dedup keepempty=t A B
http://docs.splunk.com/Documentation/Splunk/6.2.2/SearchReference/Dedup

My understanding is that dedup on 3 fields finds all matches on any two of them as duplicates. I will cite my source for that in a moment or just provide the results of a test case in support of that assertion, but I remember learning it in a Splunk course and testing it myself for validation.