topic Re: Compare rows within a table for duplicates in Splunk Search

Compare rows within a table for duplicates

rlautman — Tue, 02 Apr 2013 14:09:44 GMT

Hi,

I have created a report that takes a lookup list of order references and returns all other orders that are related, my problem is that a lot of the results from my search are duplicates - I cannot use a dedup command on any of the fields as I may miss some results - so what I wanted to do was to compare 2 fields within the rows - e.g. OrderID and CompletionStatus - check that there is no row with duplicated information, and remove 1 of the rows if there is. I have seen a similar issue here but this looks like it depends on there being only 2 rows. Is what I am asking possible and if so can anyone suggest how I would go about doing this?

Re: Compare rows within a table for duplicates

kristian_kolb — Tue, 02 Apr 2013 15:44:51 GMT

Could the combination of multikv and dedup not be of help?

From the docs on multikv:
Extracts fields from events with information in a tabular format (e.g. top, netstat, ps, ... etc). A new event will be created for each table row. Field names will be derived from the title row of the table.

Then use | dedup field1 field2 field3 to only keep unique combinations of the three fields' values.

Re: Compare rows within a table for duplicates

gkanapathy — Wed, 03 Apr 2013 22:35:46 GMT

I guess I don't see why you can't use

... | dedup OrderID CompletionStatus

which will keep only one of each combination?

Re: Compare rows within a table for duplicates

rlautman — Fri, 05 Apr 2013 09:54:24 GMT

This seems to do the job, thanks, I wasn't sure if dedup could be used for multiple fields 🙂