topic How to remove Duplicate values in different field? in Splunk Search

How to remove Duplicate values in different field?

alexspunkshell — Fri, 12 Aug 2022 15:06:40 GMT

How to remove duplicate values in a different field

|stats count by src dest

Re: Remove Duplicate values in different field

richgalloway — Thu, 11 Aug 2022 19:13:57 GMT

Where are the duplicates? I see the first 3 octets of some IP addresses match, but stats looks at the entire field, not just parts of it. If you need to deduplicate on the first 3 octets, then use rex or split to extract it into a new field and dedup on that new field.

Re: Remove Duplicate values in different field

alexspunkshell — Thu, 11 Aug 2022 19:17:52 GMT

@richgalloway Thanks for the reply

I am getting similar IPs in both src & dest fields.

So I want to remove in results if both src & dest are the same.

Re: Remove Duplicate values in different field

Taruchit — Thu, 11 Aug 2022 19:44:36 GMT

Hi @alexspunkshell,

Please confirm if I understood your requirement correctly: -

There are two fields in the result: - src and dest.

If in a given row both src and dest are same, then you need to filter out those rows from the result.

Thank you

Re: Remove Duplicate values in different field

alexspunkshell — Thu, 11 Aug 2022 19:45:52 GMT

@Taruchit You are right.

Re: Remove Duplicate values in different field

yuanliu — Fri, 12 Aug 2022 03:18:27 GMT

The most straightforward implementation is

| stats count by src dest | where src != dest

Alternatively,

| where src != dest | stats count by src dest