Splunk Search

Why is dedup not working with this search?

ErraticIncome93
Explorer

When I run the following search, I get 100+ results of src_ip 1.2.3.4 and signature X:

index=http status=200 src_ip!=10.0.0.0/8 src_ip!=192.168.0.0/16 src_ip!=172.16.0.0/12 | table src_ip | join [search index=snort | dedup src_ip,signature | table src_ip,signature]

However, if I run the search below, dedup works as expected and the combination of src_ip 1.2.3.4 and signature X only shows up once

index=snort src_ip=1.2.3.4 | dedup src_ip,signature | table src_ip signature
Tags (3)
0 Karma
1 Solution

woodcock
Esteemed Legend

It is because you are joining the first dataset (which has multiples) with the second one (which is deduped). Try this:

index=http status=200 src_ip!=10.0.0.0/8 src_ip!=192.168.0.0/16 src_ip!=172.16.0.0/12 | table src_ip | join [search index=snort] | dedup src_ip,signature | table src_ip,signature

P.S. You would not be having this trouble if you had used my stats-based solution to your previous question! 😆

View solution in original post

vasanthmss
Motivator

Hi,

index=http status=200 src_ip!=10.0.0.0/8 src_ip!=192.168.0.0/16 src_ip!=172.16.0.0/12 | table src_ip

The above query will gives you all the possible src_ip details (including duplicate entries). Then you are left joining with other index(duplicate remove).

So either you need to remove the duplicate before or after join. so the query will be like this,

index=http status=200 src_ip!=10.0.0.0/8 src_ip!=192.168.0.0/16 src_ip!=172.16.0.0/12 | dedup src_ip | table src_ip | join [search index=snort | dedup src_ip,signature | table src_ip,signature]

Thanks,
V

V

NOUMSSI
Builder

hi try this:

index=http status=200 src_ip!=10.0.0.0/8 src_ip!=192.168.0.0/16 src_ip!=172.16.0.0/12 | dedup src_ip| join [search index=snort | dedup signature | table src_ip,signature]
0 Karma

ngatchasandra
Builder

Hi Erra,

try with

index=http status=200 src_ip!=10.0.0.0/8 src_ip!=192.168.0.0/16 src_ip!=172.16.0.0/12 | table src_ip | join [search index=snort | table src_ip,signature | dedup src_ip,signature ]

0 Karma

woodcock
Esteemed Legend

It is because you are joining the first dataset (which has multiples) with the second one (which is deduped). Try this:

index=http status=200 src_ip!=10.0.0.0/8 src_ip!=192.168.0.0/16 src_ip!=172.16.0.0/12 | table src_ip | join [search index=snort] | dedup src_ip,signature | table src_ip,signature

P.S. You would not be having this trouble if you had used my stats-based solution to your previous question! 😆

ErraticIncome93
Explorer

I was working my way through both answers 😃

thanks for the help!!

0 Karma
Get Updates on the Splunk Community!

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...

Introducing Ingest Actions: Filter, Mask, Route, Repeat

WATCH NOW Ingest Actions (IA) is the best new way to easily filter, mask and route your data in Splunk® ...