Splunk Search

How to remove duplicates of one field per another field

Krapht
Explorer

Going to be very tough to explain but I'll give it my best shot. I have some fields I'm trying to report on, IP and ID.  There can be multiple duplicate ID's per IP, and vice versa. I would like to remove duplicate ID's per IP, but can't dedup on ID because some IP's could have the same ID. I also tried stats values(ID) by IP, but there are other fields that also need to be reported on and from my research I couldn't find a way to use multiple values.

Example:

What I currently get

IP1     ID1

IP1     ID1

IP1     ID2

IP1     ID2

IP2     ID1

IP2     ID1

IP2     ID2

IP2     ID2

 

What I want to get

IP1     ID1

IP1     ID2

IP2     ID1

IP2     ID2

 

OR (Preferably) in table format

IP 1    ID1     Name

            ID2     Name

-------------------------|

IP 2    ID1     Name

            ID2     Name

 

 

Labels (5)
0 Karma
1 Solution

aasabatini
Motivator

Hi @Krapht 

Can you try this?

| stats values(ID) as ID values(name) as name by IP

 Regards

Alessandro

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”

View solution in original post

0 Karma

Krapht
Explorer

I believe I found my own answer in the docs upon further research:

 

Keep results that have the same combination of values in multiple fields
For search results that have the same source AND host values, keep the first 2 that occur and remove all subsequent results.

... | dedup 2 source host

0 Karma

yuanliu
SplunkTrust
SplunkTrust

In this case, you should accept your own reply to mark the question as answered. (Yes, dedup is an effective way to do this.)

0 Karma

aasabatini
Motivator

Hi @Krapht 

Can you try this?

| stats values(ID) as ID values(name) as name by IP

 Regards

Alessandro

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”
0 Karma

Krapht
Explorer

This worked great, thanks 🙂

0 Karma
Get Updates on the Splunk Community!

See your relevant APM services, dashboards, and alerts in one place with the updated ...

As a Splunk Observability user, you have a lot of data you have to manage, prioritize, and troubleshoot on a ...

Index This | What goes away as soon as you talk about it?

May 2025 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this month’s ...

What's New in Splunk Observability Cloud and Splunk AppDynamics - May 2025

This month, we’re delivering several new innovations in Splunk Observability Cloud and Splunk AppDynamics ...