how to display unique values only from one particular multi-value field

[Umesh_Vedicsoft]

Hi
i have a field like msg="this is from: 101,102,103,101,104,102,103,105,106" but i would like to display that field with unique numbers only, without duplicates, like this msg="this is from: 101,102,103,104,105,106" using a search query
Please help!

[gcusello]

insert in your search

| rex field=msg "(?<a>\d+)" max_match=10 
| mvexpand a 
| dedup a 
| mvcombine delim=", " a 
| nomv a 
| eval msg="this is from: "+a 
| table msg

Bye.
Giuseppe

[javiergn]

Option with stats values (Assuming there is _raw field in your data 😞

your base search here
| rex field=msg "^(?<msg_text>.+?)(?<msg_numbers>[\d\,]+)$"
| eval msg_numbers = split(msg_numbers, ",")
| mvexpand msg_numbers
| stats values(msg_numbers) as msg_numbers by _raw, msg_text
| eval msg_numbers = mvjoin(msg_numbers, ",")
| eval msg = msg_text . msg_numbers

Another option with mvdedup:

your base search here
| rex field=msg "^(?<msg_text>.+?)(?<msg_numbers>[\d\,]+)$"
| eval msg_numbers = mvjoin(mvdedup(split(msg_numbers, ",")), ",")
| eval msg = msg_text . msg_numbers

Example:

| stats count | fields - count
| eval _raw = "_time=2016-12-18 00:00:00 this is from: 101,102,103,101,104,102,103,105,106"
| eval msg = "this is from: 101,102,103,101,104,102,103,105,106"
| rex field=msg "^(?<msg_text>.+?)(?<msg_numbers>[\d\,]+)$"
| eval msg_numbers = split(msg_numbers, ",")
| mvexpand msg_numbers
| stats values(msg_numbers) as msg_numbers by _raw, msg_text
| eval msg_numbers = mvjoin(msg_numbers, ",")
| eval msg = msg_text . msg_numbers

Output (see picture below):

[hgrow]

Hi there,

maybe the eval-statement mvdedup is what you are looking for.

You can just execute the query below as an example:

| stats count | eval msg="this is from: 101,102,103,101,104,102,103,105,106" | rex max_match=99 field=msg "(?<msg_from>\d+" | eval u_msg_from=mvdedup(msg_from)

Greetings
hgrow

[arkadyz1]

Or, if you are certain the original field has only this format, something like:

| rex field=msg "^this is from: (?P<msg_from>.*)$" | eval msg_from=split(msg_from, ",") | eval msg_from=mvdedup(msg_from) | eval msg_from=mvjoin(msg_from, ",")

I'm not sure which way is more efficient - rex with max_match or split, but it is always nice to have more than one way of doing things.