Splunk Search

How to remove repeated duplicate values?

alexspunkshell
Contributor

I am trying to remove duplicates in my result using the |dedup command. Even though I am seeing 2 entries in my result.

Kindly help me to remove 1 duplicate.

 

alexspunkshell_0-1633093094982.png

 

Labels (2)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust
| stats count by title
| fields - count

View solution in original post

Lavender
Loves-to-Learn Everything

I too having same kind of issue . I have tried with your solution , but still I can see duplicate values . please help @ITWhisperer 

 

Lavender_0-1681983560162.png

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

I have no idea what SPL you used to create that output since you didn't share it. Having said that, given that you apparently have three occurrences of the same string, perhaps your base data is at fault, or perhaps you have trailing spaces?

0 Karma

Lavender
Loves-to-Learn Everything

I got the solution . 

mvexpand doesn't work because the field is not a multi-value field.  It's a single-value field with embedded newlines.  Tried using the split function to break up the field then mvexpand  and it works

0 Karma

Azeemering
Builder

This field contains multiple duplicate values I guess.

You can remove it like this:

index=graphsecurityalert | mvexpand title | dedup title | table title

Check to see which events contain those multivalues:

index=graphsecurityalert | eval c=mvcount(title) | table c

 

ITWhisperer
SplunkTrust
SplunkTrust
| eval title=mvdedup(title)

ITWhisperer
SplunkTrust
SplunkTrust
| stats count by title
| fields - count
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...