Splunk Search

How to remove duplicate values of a field inside of a row/column?

sina_shafaei
Explorer

Hi

I have a table like this (there are other fields between first and last field):

Var1-------...-------Varn

First              AB
                   BA
                   CA
                   BA
                   C 

Second             BA
                   AB
                   CD

Last               CF
                   BA
                   BA
                   CF

as you can see in some rows (first and last) values for Var2 have some duplicates (e.g. in first row value BA is duplicated, and also in last one CF is the same) I want to remove the second duplicate values in each row for second column(Var2) and also remove c from first row to be like this at the end:

Var1-------...-------Varn

First              AB
                   BA
                   CA


Second             BA
                   AB
                   CD

Last               CF
                   BA

can you please help me and tell me how can I do that?
I need a general way that works for lot's of rows (checking around 256 rows to find duplicates and remove them)

Thank you so much!

Cheers

0 Karma

strive
Influencer

What is the condition under which you should remove C?

To answer your question, i considered a condition that length of the value should be greater than 1.

Try this

Your Base Search.. | stats values(VarN) as VarN by Var1 | eval VarN = mvfilter(len(VarN)>1)

values(VarN) gives you the distinct values. mvfilter can be used to filter the values of a multivalue field.

strive
Influencer

Those other columns will they contain single value or they also contain multiple values?

Assuming they contain single values

Your Base Search.. | stats first(Var2) as Var2, first(Var3) as Var3, values(VarN) as VarN by Var1 | eval VarN = mvfilter(len(VarN)>1)

0 Karma

sina_shafaei
Explorer

what if I want to add another columns? because in this case I have another parameters which I wanted to add beside Var1 and Varn (like Var2,...) is there any way to use table for this purpose? not list?

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...