Splunk Search

How to remove duplicates of one field per another field

Krapht
Explorer

Going to be very tough to explain but I'll give it my best shot. I have some fields I'm trying to report on, IP and ID.  There can be multiple duplicate ID's per IP, and vice versa. I would like to remove duplicate ID's per IP, but can't dedup on ID because some IP's could have the same ID. I also tried stats values(ID) by IP, but there are other fields that also need to be reported on and from my research I couldn't find a way to use multiple values.

Example:

What I currently get

IP1     ID1

IP1     ID1

IP1     ID2

IP1     ID2

IP2     ID1

IP2     ID1

IP2     ID2

IP2     ID2

 

What I want to get

IP1     ID1

IP1     ID2

IP2     ID1

IP2     ID2

 

OR (Preferably) in table format

IP 1    ID1     Name

            ID2     Name

-------------------------|

IP 2    ID1     Name

            ID2     Name

 

 

Labels (5)
0 Karma
1 Solution

aasabatini
Motivator

Hi @Krapht 

Can you try this?

| stats values(ID) as ID values(name) as name by IP

 Regards

Alessandro

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”

View solution in original post

0 Karma

Krapht
Explorer

I believe I found my own answer in the docs upon further research:

 

Keep results that have the same combination of values in multiple fields
For search results that have the same source AND host values, keep the first 2 that occur and remove all subsequent results.

... | dedup 2 source host

0 Karma

yuanliu
SplunkTrust
SplunkTrust

In this case, you should accept your own reply to mark the question as answered. (Yes, dedup is an effective way to do this.)

0 Karma

aasabatini
Motivator

Hi @Krapht 

Can you try this?

| stats values(ID) as ID values(name) as name by IP

 Regards

Alessandro

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”
0 Karma

Krapht
Explorer

This worked great, thanks 🙂

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Leveraging Automated Threat Analysis Across the Splunk Ecosystem

Are you leveraging automation to its fullest potential in your threat detection strategy?Our upcoming Security ...

Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain

Boston may be buzzing this September with Splunk University and .conf25, but you don’t have to pack a bag to ...

Splunk Lantern’s Guide to The Most Popular .conf25 Sessions

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...