topic Re: How to remove duplicates of one field per another field in Splunk Search

How to remove duplicates of one field per another field

Krapht — Tue, 25 May 2021 05:53:20 GMT

Going to be very tough to explain but I'll give it my best shot. I have some fields I'm trying to report on, IP and ID. There can be multiple duplicate ID's per IP, and vice versa. I would like to remove duplicate ID's per IP, but can't dedup on ID because some IP's could have the same ID. I also tried stats values(ID) by IP, but there are other fields that also need to be reported on and from my research I couldn't find a way to use multiple values.

Example:

What I currently get

IP1 ID1

IP1 ID2

IP2 ID1

IP2 ID2

What I want to get

IP1 ID1

IP1 ID2

IP2 ID1

IP2 ID2

OR (Preferably) in table format

IP 1 ID1 Name

ID2 Name

-------------------------|

IP 2 ID1 Name

ID2 Name

Re: How to remove duplicates of one field per another field

aasabatini — Tue, 25 May 2021 06:03:50 GMT

Hi @Krapht

Can you try this?

| stats values(ID) as ID values(name) as name by IP

Regards

Alessandro

Re: How to remove duplicates of one field per another field

Krapht — Tue, 25 May 2021 06:48:56 GMT

I believe I found my own answer in the docs upon further research:

Keep results that have the same combination of values in multiple fields
For search results that have the same source AND host values, keep the first 2 that occur and remove all subsequent results.

... | dedup 2 source host

Re: How to remove duplicates of one field per another field

yuanliu — Tue, 25 May 2021 07:23:50 GMT

In this case, you should accept your own reply to mark the question as answered. (Yes, dedup is an effective way to do this.)

Re: How to remove duplicates of one field per another field

Krapht — Tue, 25 May 2021 15:59:08 GMT

This worked great, thanks 🙂