Splunk Search

how to display unique values only from one particular multi-value field

Path Finder

Hi
i have a field like msg="this is from: 101,102,103,101,104,102,103,105,106" but i would like to display that field with unique numbers only, without duplicates, like this msg="this is from: 101,102,103,104,105,106" using a search query
Please help!

0 Karma

SplunkTrust
SplunkTrust

insert in your search

| rex field=msg "(?<a>\d+)" max_match=10 
| mvexpand a 
| dedup a 
| mvcombine delim=", " a 
| nomv a 
| eval msg="this is from: "+a 
| table msg

Bye.
Giuseppe

0 Karma

SplunkTrust
SplunkTrust

Option with stats values (Assuming there is _raw field in your data 😞

your base search here
| rex field=msg "^(?<msg_text>.+?)(?<msg_numbers>[\d\,]+)$"
| eval msg_numbers = split(msg_numbers, ",")
| mvexpand msg_numbers
| stats values(msg_numbers) as msg_numbers by _raw, msg_text
| eval msg_numbers = mvjoin(msg_numbers, ",")
| eval msg = msg_text . msg_numbers

Another option with mvdedup:

your base search here
| rex field=msg "^(?<msg_text>.+?)(?<msg_numbers>[\d\,]+)$"
| eval msg_numbers = mvjoin(mvdedup(split(msg_numbers, ",")), ",")
| eval msg = msg_text . msg_numbers

Example:

| stats count | fields - count
| eval _raw = "_time=2016-12-18 00:00:00 this is from: 101,102,103,101,104,102,103,105,106"
| eval msg = "this is from: 101,102,103,101,104,102,103,105,106"
| rex field=msg "^(?<msg_text>.+?)(?<msg_numbers>[\d\,]+)$"
| eval msg_numbers = split(msg_numbers, ",")
| mvexpand msg_numbers
| stats values(msg_numbers) as msg_numbers by _raw, msg_text
| eval msg_numbers = mvjoin(msg_numbers, ",")
| eval msg = msg_text . msg_numbers

Output (see picture below):

alt text

0 Karma

Communicator

Hi there,

maybe the eval-statement mvdedup is what you are looking for.

You can just execute the query below as an example:

| stats count | eval msg="this is from: 101,102,103,101,104,102,103,105,106" | rex max_match=99 field=msg "(?<msg_from>\d+" | eval u_msg_from=mvdedup(msg_from)

Greetings
hgrow

Builder

Or, if you are certain the original field has only this format, something like:

| rex field=msg "^this is from: (?P<msg_from>.*)$" | eval msg_from=split(msg_from, ",") | eval msg_from=mvdedup(msg_from) | eval msg_from=mvjoin(msg_from, ",")

I'm not sure which way is more efficient - rex with max_match or split, but it is always nice to have more than one way of doing things.

State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!