Splunk Search

How to remove duplicates of one field per another field

Krapht
Explorer

Going to be very tough to explain but I'll give it my best shot. I have some fields I'm trying to report on, IP and ID.  There can be multiple duplicate ID's per IP, and vice versa. I would like to remove duplicate ID's per IP, but can't dedup on ID because some IP's could have the same ID. I also tried stats values(ID) by IP, but there are other fields that also need to be reported on and from my research I couldn't find a way to use multiple values.

Example:

What I currently get

IP1     ID1

IP1     ID1

IP1     ID2

IP1     ID2

IP2     ID1

IP2     ID1

IP2     ID2

IP2     ID2

 

What I want to get

IP1     ID1

IP1     ID2

IP2     ID1

IP2     ID2

 

OR (Preferably) in table format

IP 1    ID1     Name

            ID2     Name

-------------------------|

IP 2    ID1     Name

            ID2     Name

 

 

Labels (5)
0 Karma
1 Solution

aasabatini
Motivator

Hi @Krapht 

Can you try this?

| stats values(ID) as ID values(name) as name by IP

 Regards

Alessandro

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”

View solution in original post

0 Karma

Krapht
Explorer

I believe I found my own answer in the docs upon further research:

 

Keep results that have the same combination of values in multiple fields
For search results that have the same source AND host values, keep the first 2 that occur and remove all subsequent results.

... | dedup 2 source host

0 Karma

yuanliu
SplunkTrust
SplunkTrust

In this case, you should accept your own reply to mark the question as answered. (Yes, dedup is an effective way to do this.)

0 Karma

aasabatini
Motivator

Hi @Krapht 

Can you try this?

| stats values(ID) as ID values(name) as name by IP

 Regards

Alessandro

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”
0 Karma

Krapht
Explorer

This worked great, thanks 🙂

0 Karma
Get Updates on the Splunk Community!

OpenTelemetry for Legacy Apps? Yes, You Can!

This article is a follow-up to my previous article posted on the OpenTelemetry Blog, "Your Critical Legacy App ...

UCC Framework: Discover Developer Toolkit for Building Technology Add-ons

The Next-Gen Toolkit for Splunk Technology Add-on Development The Universal Configuration Console (UCC) ...

.conf25 Community Recap

Hello Splunkers, And just like that, .conf25 is in the books! What an incredible few days — full of learning, ...