topic Re: How to remove repeated duplicate values in Splunk Search

How to remove repeated duplicate values?

alexspunkshell — Thu, 20 Apr 2023 16:12:24 GMT

I am trying to remove duplicates in my result using the |dedup command. Even though I am seeing 2 entries in my result.

Kindly help me to remove 1 duplicate.

Re: How to remove repeated duplicate values

ITWhisperer — Fri, 01 Oct 2021 13:00:39 GMT

| stats count by title | fields - count

Re: How to remove repeated duplicate values

ITWhisperer — Fri, 01 Oct 2021 13:01:39 GMT

| eval title=mvdedup(title)

Re: How to remove repeated duplicate values

Azeemering — Fri, 01 Oct 2021 13:09:42 GMT

This field contains multiple duplicate values I guess.

You can remove it like this:

index=graphsecurityalert | mvexpand title | dedup title | table title

Check to see which events contain those multivalues:

index=graphsecurityalert | eval c=mvcount(title) | table c

Re: How to remove repeated duplicate values

Lavender — Thu, 20 Apr 2023 09:40:28 GMT

I too having same kind of issue . I have tried with your solution , but still I can see duplicate values . please help @ITWhisperer

Re: How to remove repeated duplicate values

ITWhisperer — Thu, 20 Apr 2023 09:46:29 GMT

I have no idea what SPL you used to create that output since you didn't share it. Having said that, given that you apparently have three occurrences of the same string, perhaps your base data is at fault, or perhaps you have trailing spaces?

Re: How to remove repeated duplicate values

Lavender — Thu, 20 Apr 2023 10:06:49 GMT

I got the solution .

mvexpand doesn't work because the field is not a multi-value field. It's a single-value field with embedded newlines. Tried using the split function to break up the field then mvexpand and it works