Splunk Search
Highlighted

Why is dedup not working with this search?

When I run the following search, I get 100+ results of src_ip 1.2.3.4 and signature X:

index=http status=200 src_ip!=10.0.0.0/8 src_ip!=192.168.0.0/16 src_ip!=172.16.0.0/12 | table src_ip | join [search index=snort | dedup src_ip,signature | table src_ip,signature]

However, if I run the search below, dedup works as expected and the combination of src_ip 1.2.3.4 and signature X only shows up once

index=snort src_ip=1.2.3.4 | dedup src_ip,signature | table src_ip signature
Tags (3)
0 Karma
Highlighted

Re: Why is dedup not working with this search?

Esteemed Legend

It is because you are joining the first dataset (which has multiples) with the second one (which is deduped). Try this:

index=http status=200 src_ip!=10.0.0.0/8 src_ip!=192.168.0.0/16 src_ip!=172.16.0.0/12 | table src_ip | join [search index=snort] | dedup src_ip,signature | table src_ip,signature

P.S. You would not be having this trouble if you had used my stats-based solution to your previous question! 😆

View solution in original post

Highlighted

Re: Why is dedup not working with this search?

I was working my way through both answers 😃

thanks for the help!!

0 Karma
Highlighted

Re: Why is dedup not working with this search?

Hi Erra,

try with

index=http status=200 srcip!=10.0.0.0/8 srcip!=192.168.0.0/16 srcip!=172.16.0.0/12 | table srcip | join [search index=snort | table srcip,signature | dedup srcip,signature ]

0 Karma
Highlighted

Re: Why is dedup not working with this search?

Builder

hi try this:

index=http status=200 src_ip!=10.0.0.0/8 src_ip!=192.168.0.0/16 src_ip!=172.16.0.0/12 | dedup src_ip| join [search index=snort | dedup signature | table src_ip,signature]
0 Karma
Highlighted

Re: Why is dedup not working with this search?

Motivator

Hi,

index=http status=200 src_ip!=10.0.0.0/8 src_ip!=192.168.0.0/16 src_ip!=172.16.0.0/12 | table src_ip

The above query will gives you all the possible src_ip details (including duplicate entries). Then you are left joining with other index(duplicate remove).

So either you need to remove the duplicate before or after join. so the query will be like this,

index=http status=200 src_ip!=10.0.0.0/8 src_ip!=192.168.0.0/16 src_ip!=172.16.0.0/12 | dedup src_ip | table src_ip | join [search index=snort | dedup src_ip,signature | table src_ip,signature]

Thanks,
V