Remove the duplicate values from each row of "FINAL" filed.

[abhayneilam]

I have a column called "FINAL" which contains :

FINAL

---

kolkata,mumbai,kolkata
delhi,mumbai
delhi,delhi
hydrabad
hydrabad

My output should be :

FINAL

---

kolkata,mumbai
delhi,mumbai
delhi
hydrabad
hydrabad

I just want to remove the duplicate value from each row of "FINAL" filed..

Please help !!

Thanks in Advance,
Abhay

[all caps title edited]

[yannK]

So If the FINAL field has multiple values on each event


event1 : FINAL="kolkata,mumbai"
event2 : FINAL="delhi,mumbai"
event3 : FINAL="delhi"
event4 : FINAL="hydrabad"
event5 : FINAL="hydrabad"


Then you should make sure that the fields contains multivalues, if not, make sure to extract them.

Here is a way to remoce the comma, then remove the duplicated per event, and recombine the multivalue event.

* | makemv delim="," FINAL
| stats values(FINAL) AS FINAL by _raw
| nomv FINAL
| table FINAL


[Simon]

Have a look at the "dedup" search command (http://docs.splunk.com/Documentation/Splunk/5.0/SearchReference/dedup):

your search | fields FINAL | dedup FINAL 

//edit: Sorry, I didn't see that you wan't remove duplicate inside the field. Is it a multi-value field or just a simple string? Do you concatenate the FINAL field first? Maybe you could use dedup before concatenating.

[abhayneilam]

This is a multi-value string not a simple string...Please help me achieve this !!

Thanks in Advance
Abhay