topic Re: Dedup within a MV field in Splunk Search

Dedup within a MV field

pkashou — Fri, 15 Feb 2013 23:00:55 GMT

I need the ability to dedup a multi-value field on a per event basis. Something like values() but limited to one event at a time. The ordering within the mv doesn't matter to me, just that there aren't duplicates. Any help is greatly appreciated.

My search:

Normal output:

Preferred output:

Re: Dedup within a MV field

martin_mueller — Fri, 15 Feb 2013 23:19:50 GMT

You could make use of the regular dedup like this:

...  | streamstats count | mvexpand eventSplit | dedup count eventSplit | mvcombine eventSplit | fields - count

Re: Dedup within a MV field

sideview — Sat, 16 Feb 2013 00:45:53 GMT

Another idea is to use stats values(), but do a weird trick to make it calculate unique values only within each row.

| streamstats count as row_number | stats values(mvField) as mvField by row_number | fields - row_number

Re: Dedup within a MV field

pkashou — Sun, 17 Feb 2013 16:51:51 GMT

Thanks to both of you as these both worked to a certain degree. The stats weird trick did some strangeness to the output so I ended up using the mvexpand/mvcombine approach along with eventstats.

Much appreciated!

Re: Dedup within a MV field

emiller42 — Tue, 10 Feb 2015 15:33:52 GMT

I know this is an old question, but I stumbled upon this while trying to do the same thing, and there is now a much cleaner solution:

eval mvfield=mvdedup(mvfield)

Re: Dedup within a MV field

redc — Tue, 19 May 2015 19:48:34 GMT

I ran into this need today and stumbled across this post...

It's worth noting for anyone else who finds this post while trying to figure out how to do this that <code>mvdedup</code> was only introduced in 6.2.0.

Re: Dedup within a MV field

danbutterman — Tue, 12 Dec 2017 21:06:05 GMT

Exactly what I was looking for.

Love this community.