Splunk Search

How to remove repeated duplicate values?

alexspunkshell
Contributor

I am trying to remove duplicates in my result using the |dedup command. Even though I am seeing 2 entries in my result.

Kindly help me to remove 1 duplicate.

 

alexspunkshell_0-1633093094982.png

 

Labels (2)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust
| stats count by title
| fields - count

View solution in original post

Lavender
Loves-to-Learn Everything

I too having same kind of issue . I have tried with your solution , but still I can see duplicate values . please help @ITWhisperer 

 

Lavender_0-1681983560162.png

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

I have no idea what SPL you used to create that output since you didn't share it. Having said that, given that you apparently have three occurrences of the same string, perhaps your base data is at fault, or perhaps you have trailing spaces?

0 Karma

Lavender
Loves-to-Learn Everything

I got the solution . 

mvexpand doesn't work because the field is not a multi-value field.  It's a single-value field with embedded newlines.  Tried using the split function to break up the field then mvexpand  and it works

0 Karma

Azeemering
Builder

This field contains multiple duplicate values I guess.

You can remove it like this:

index=graphsecurityalert | mvexpand title | dedup title | table title

Check to see which events contain those multivalues:

index=graphsecurityalert | eval c=mvcount(title) | table c

 

ITWhisperer
SplunkTrust
SplunkTrust
| eval title=mvdedup(title)

ITWhisperer
SplunkTrust
SplunkTrust
| stats count by title
| fields - count
Get Updates on the Splunk Community!

Leveraging Automated Threat Analysis Across the Splunk Ecosystem

Enhance Security Operations with Automated Threat Analysis in the Splunk EcosystemAre you leveraging ...

Splunk Developers: Go Beyond the Dashboard with These .Conf25 Sessions

  Whether you’re building custom apps, diving into SPL2, or integrating AI and machine learning into your ...

Index This | How do you write 23 only using the number 2?

July 2025 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this month’s ...