Splunk Search

How to remove duplicates of one field per another field

Krapht
Explorer

Going to be very tough to explain but I'll give it my best shot. I have some fields I'm trying to report on, IP and ID.  There can be multiple duplicate ID's per IP, and vice versa. I would like to remove duplicate ID's per IP, but can't dedup on ID because some IP's could have the same ID. I also tried stats values(ID) by IP, but there are other fields that also need to be reported on and from my research I couldn't find a way to use multiple values.

Example:

What I currently get

IP1     ID1

IP1     ID1

IP1     ID2

IP1     ID2

IP2     ID1

IP2     ID1

IP2     ID2

IP2     ID2

 

What I want to get

IP1     ID1

IP1     ID2

IP2     ID1

IP2     ID2

 

OR (Preferably) in table format

IP 1    ID1     Name

            ID2     Name

-------------------------|

IP 2    ID1     Name

            ID2     Name

 

 

Labels (5)
0 Karma
1 Solution

aasabatini
Motivator

Hi @Krapht 

Can you try this?

| stats values(ID) as ID values(name) as name by IP

 Regards

Alessandro

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”

View solution in original post

0 Karma

Krapht
Explorer

I believe I found my own answer in the docs upon further research:

 

Keep results that have the same combination of values in multiple fields
For search results that have the same source AND host values, keep the first 2 that occur and remove all subsequent results.

... | dedup 2 source host

0 Karma

yuanliu
SplunkTrust
SplunkTrust

In this case, you should accept your own reply to mark the question as answered. (Yes, dedup is an effective way to do this.)

0 Karma

aasabatini
Motivator

Hi @Krapht 

Can you try this?

| stats values(ID) as ID values(name) as name by IP

 Regards

Alessandro

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”
0 Karma

Krapht
Explorer

This worked great, thanks 🙂

0 Karma
Get Updates on the Splunk Community!

Developer Spotlight with William Searle

The Splunk Guy: A Developer’s Path from Web to Cloud William is a Splunk Professional Services Consultant with ...

Major Splunk Upgrade – Prepare your Environment for Splunk 10 Now!

Attention App Developers: Test Your Apps with the Splunk 10.0 Beta and Ensure Compatibility Before the ...

Stay Connected: Your Guide to June Tech Talks, Office Hours, and Webinars!

What are Community Office Hours?Community Office Hours is an interactive 60-minute Zoom series where ...