Splunk Search

How to remove duplicates of one field per another field

Krapht
Explorer

Going to be very tough to explain but I'll give it my best shot. I have some fields I'm trying to report on, IP and ID.  There can be multiple duplicate ID's per IP, and vice versa. I would like to remove duplicate ID's per IP, but can't dedup on ID because some IP's could have the same ID. I also tried stats values(ID) by IP, but there are other fields that also need to be reported on and from my research I couldn't find a way to use multiple values.

Example:

What I currently get

IP1     ID1

IP1     ID1

IP1     ID2

IP1     ID2

IP2     ID1

IP2     ID1

IP2     ID2

IP2     ID2

 

What I want to get

IP1     ID1

IP1     ID2

IP2     ID1

IP2     ID2

 

OR (Preferably) in table format

IP 1    ID1     Name

            ID2     Name

-------------------------|

IP 2    ID1     Name

            ID2     Name

 

 

Labels (5)
0 Karma
1 Solution

aasabatini
Motivator

Hi @Krapht 

Can you try this?

| stats values(ID) as ID values(name) as name by IP

 Regards

Alessandro

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”

View solution in original post

0 Karma

Krapht
Explorer

I believe I found my own answer in the docs upon further research:

 

Keep results that have the same combination of values in multiple fields
For search results that have the same source AND host values, keep the first 2 that occur and remove all subsequent results.

... | dedup 2 source host

0 Karma

yuanliu
SplunkTrust
SplunkTrust

In this case, you should accept your own reply to mark the question as answered. (Yes, dedup is an effective way to do this.)

0 Karma

aasabatini
Motivator

Hi @Krapht 

Can you try this?

| stats values(ID) as ID values(name) as name by IP

 Regards

Alessandro

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”
0 Karma

Krapht
Explorer

This worked great, thanks 🙂

0 Karma
Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Dynamic formatting from XML events

This challenge was first posted on Slack #puzzles channelFor a previous puzzle, I needed a set of fixed-length ...

Enter the Agentic Era with Splunk AI Assistant for SPL 1.4

  🚀 Your data just got a serious AI upgrade — are you ready? Say hello to the Agentic Era with the ...

Stronger Security with Federated Search for S3, GCP SQL & Australian Threat ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...