Splunk Search

How to remove duplicate values of a field inside of a row/column?

sina_shafaei
Explorer

Hi

I have a table like this (there are other fields between first and last field):

Var1-------...-------Varn

First              AB
                   BA
                   CA
                   BA
                   C 

Second             BA
                   AB
                   CD

Last               CF
                   BA
                   BA
                   CF

as you can see in some rows (first and last) values for Var2 have some duplicates (e.g. in first row value BA is duplicated, and also in last one CF is the same) I want to remove the second duplicate values in each row for second column(Var2) and also remove c from first row to be like this at the end:

Var1-------...-------Varn

First              AB
                   BA
                   CA


Second             BA
                   AB
                   CD

Last               CF
                   BA

can you please help me and tell me how can I do that?
I need a general way that works for lot's of rows (checking around 256 rows to find duplicates and remove them)

Thank you so much!

Cheers

0 Karma

strive
Influencer

What is the condition under which you should remove C?

To answer your question, i considered a condition that length of the value should be greater than 1.

Try this

Your Base Search.. | stats values(VarN) as VarN by Var1 | eval VarN = mvfilter(len(VarN)>1)

values(VarN) gives you the distinct values. mvfilter can be used to filter the values of a multivalue field.

strive
Influencer

Those other columns will they contain single value or they also contain multiple values?

Assuming they contain single values

Your Base Search.. | stats first(Var2) as Var2, first(Var3) as Var3, values(VarN) as VarN by Var1 | eval VarN = mvfilter(len(VarN)>1)

0 Karma

sina_shafaei
Explorer

what if I want to add another columns? because in this case I have another parameters which I wanted to add beside Var1 and Varn (like Var2,...) is there any way to use table for this purpose? not list?

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...