topic Re: How to remove duplicate column values in table? in Splunk Search

How to remove duplicate column values in table?

the_wolverine — Thu, 18 Jul 2013 17:06:01 GMT

I have a table that contains several columns. The table looks something like this:

timestamp,region,product_number,status,count
time1,americas,12345,done,5
time2,americas,23456,fail,4
time3,emea,34567,done,3
time4,emea,56789,fail,1
time5,emea,56789,done,2

I would like to reformat it to remove the column values that are duplicated:

timestamp,region,product_number,status,count
time1,americas,12345,done,5
time2,        ,23456,fail,4
time3,emea,34567,done,3
time4,    ,56789,fail,1
time5,    ,     ,done,2

Re: How to remove duplicate column values in table?

Ayn — Thu, 18 Jul 2013 18:38:18 GMT

I don't think there's a generic solution to remove "duplicates" in any field (at least I can't think of one), but if it's OK to specify fields you could achieve this with streamstats.

... | streamstats current=f last(region) as _lastregion, last(product_number) as _last_product_number | eval region=if(region==_lastregion,"",region) | eval product_number=if(product_number==_last_product_number,"",product_number) | ... (and so on)

Re: How to remove duplicate column values in table?

the_wolverine — Thu, 18 Jul 2013 23:55:11 GMT

It doesn't produce the table I'm looking for unfortunately. What I get from your suggestion is:

timestamp,region,product_number,status,count
time1, ,12345,done,5
time2, ,23456,fail,4
time3,emea,34567,done,3
time4,emea,56789,fail,1
time5,emea, ,done,2

Re: How to remove duplicate column values in table?

bmacias84 — Fri, 19 Jul 2013 22:15:20 GMT

have you considered using selfjoin on region?

Re: How to remove duplicate column values in table?

the_wolverine — Wed, 24 Jul 2013 23:00:34 GMT

Um, it works. Turns out I had bad data. Thanks, Ayn!