Compare rows within a table for duplicates

[rlautman]

Hi,

I have created a report that takes a lookup list of order references and returns all other orders that are related, my problem is that a lot of the results from my search are duplicates - I cannot use a dedup command on any of the fields as I may miss some results - so what I wanted to do was to compare 2 fields within the rows - e.g. OrderID and CompletionStatus - check that there is no row with duplicated information, and remove 1 of the rows if there is. I have seen a similar issue here but this looks like it depends on there being only 2 rows. Is what I am asking possible and if so can anyone suggest how I would go about doing this?

[gkanapathy · Accepted solution]

I guess I don't see why you can't use

... | dedup OrderID CompletionStatus

which will keep only one of each combination?

[rlautman]

This seems to do the job, thanks, I wasn't sure if dedup could be used for multiple fields 🙂

[kristian_kolb]

Could the combination of multikv and dedup not be of help?

From the docs on multikv:
Extracts fields from events with information in a tabular format (e.g. top, netstat, ps, ... etc). A new event will be created for each table row. Field names will be derived from the title row of the table.

Then use | dedup field1 field2 field3 to only keep unique combinations of the three fields' values.

/K