Splunk Search

Why is dedup not working with this search?

ErraticIncome93
Explorer

When I run the following search, I get 100+ results of src_ip 1.2.3.4 and signature X:

index=http status=200 src_ip!=10.0.0.0/8 src_ip!=192.168.0.0/16 src_ip!=172.16.0.0/12 | table src_ip | join [search index=snort | dedup src_ip,signature | table src_ip,signature]

However, if I run the search below, dedup works as expected and the combination of src_ip 1.2.3.4 and signature X only shows up once

index=snort src_ip=1.2.3.4 | dedup src_ip,signature | table src_ip signature
Tags (3)
0 Karma
1 Solution

woodcock
Esteemed Legend

It is because you are joining the first dataset (which has multiples) with the second one (which is deduped). Try this:

index=http status=200 src_ip!=10.0.0.0/8 src_ip!=192.168.0.0/16 src_ip!=172.16.0.0/12 | table src_ip | join [search index=snort] | dedup src_ip,signature | table src_ip,signature

P.S. You would not be having this trouble if you had used my stats-based solution to your previous question! 😆

View solution in original post

vasanthmss
Motivator

Hi,

index=http status=200 src_ip!=10.0.0.0/8 src_ip!=192.168.0.0/16 src_ip!=172.16.0.0/12 | table src_ip

The above query will gives you all the possible src_ip details (including duplicate entries). Then you are left joining with other index(duplicate remove).

So either you need to remove the duplicate before or after join. so the query will be like this,

index=http status=200 src_ip!=10.0.0.0/8 src_ip!=192.168.0.0/16 src_ip!=172.16.0.0/12 | dedup src_ip | table src_ip | join [search index=snort | dedup src_ip,signature | table src_ip,signature]

Thanks,
V

V

NOUMSSI
Builder

hi try this:

index=http status=200 src_ip!=10.0.0.0/8 src_ip!=192.168.0.0/16 src_ip!=172.16.0.0/12 | dedup src_ip| join [search index=snort | dedup signature | table src_ip,signature]
0 Karma

ngatchasandra
Builder

Hi Erra,

try with

index=http status=200 src_ip!=10.0.0.0/8 src_ip!=192.168.0.0/16 src_ip!=172.16.0.0/12 | table src_ip | join [search index=snort | table src_ip,signature | dedup src_ip,signature ]

0 Karma

woodcock
Esteemed Legend

It is because you are joining the first dataset (which has multiples) with the second one (which is deduped). Try this:

index=http status=200 src_ip!=10.0.0.0/8 src_ip!=192.168.0.0/16 src_ip!=172.16.0.0/12 | table src_ip | join [search index=snort] | dedup src_ip,signature | table src_ip,signature

P.S. You would not be having this trouble if you had used my stats-based solution to your previous question! 😆

ErraticIncome93
Explorer

I was working my way through both answers 😃

thanks for the help!!

0 Karma
Get Updates on the Splunk Community!

Build Your First SPL2 App!

Watch the recording now!.Do you want to SPL™, too? SPL2, Splunk's next-generation data search and preparation ...

Exporting Splunk Apps

Join us on Monday, October 21 at 11 am PT | 2 pm ET!With the app export functionality, app developers and ...

[Coming Soon] Splunk Observability Cloud - Enhanced navigation with a modern look and ...

We are excited to introduce our enhanced UI that brings together AppDynamics and Splunk Observability. This is ...