Splunk Search

how to display unique values only from one particular multi-value field

Umesh_Vedicsoft
Path Finder

Hi
i have a field like msg="this is from: 101,102,103,101,104,102,103,105,106" but i would like to display that field with unique numbers only, without duplicates, like this msg="this is from: 101,102,103,104,105,106" using a search query
Please help!

0 Karma

gcusello
SplunkTrust
SplunkTrust

insert in your search

| rex field=msg "(?<a>\d+)" max_match=10 
| mvexpand a 
| dedup a 
| mvcombine delim=", " a 
| nomv a 
| eval msg="this is from: "+a 
| table msg

Bye.
Giuseppe

0 Karma

javiergn
Super Champion

Option with stats values (Assuming there is _raw field in your data 😞

your base search here
| rex field=msg "^(?<msg_text>.+?)(?<msg_numbers>[\d\,]+)$"
| eval msg_numbers = split(msg_numbers, ",")
| mvexpand msg_numbers
| stats values(msg_numbers) as msg_numbers by _raw, msg_text
| eval msg_numbers = mvjoin(msg_numbers, ",")
| eval msg = msg_text . msg_numbers

Another option with mvdedup:

your base search here
| rex field=msg "^(?<msg_text>.+?)(?<msg_numbers>[\d\,]+)$"
| eval msg_numbers = mvjoin(mvdedup(split(msg_numbers, ",")), ",")
| eval msg = msg_text . msg_numbers

Example:

| stats count | fields - count
| eval _raw = "_time=2016-12-18 00:00:00 this is from: 101,102,103,101,104,102,103,105,106"
| eval msg = "this is from: 101,102,103,101,104,102,103,105,106"
| rex field=msg "^(?<msg_text>.+?)(?<msg_numbers>[\d\,]+)$"
| eval msg_numbers = split(msg_numbers, ",")
| mvexpand msg_numbers
| stats values(msg_numbers) as msg_numbers by _raw, msg_text
| eval msg_numbers = mvjoin(msg_numbers, ",")
| eval msg = msg_text . msg_numbers

Output (see picture below):

alt text

0 Karma

hgrow
Communicator

Hi there,

maybe the eval-statement mvdedup is what you are looking for.

You can just execute the query below as an example:

| stats count | eval msg="this is from: 101,102,103,101,104,102,103,105,106" | rex max_match=99 field=msg "(?<msg_from>\d+" | eval u_msg_from=mvdedup(msg_from)

Greetings
hgrow

arkadyz1
Builder

Or, if you are certain the original field has only this format, something like:

| rex field=msg "^this is from: (?P<msg_from>.*)$" | eval msg_from=split(msg_from, ",") | eval msg_from=mvdedup(msg_from) | eval msg_from=mvjoin(msg_from, ",")

I'm not sure which way is more efficient - rex with max_match or split, but it is always nice to have more than one way of doing things.

Get Updates on the Splunk Community!

Splunk Observability Cloud's AI Assistant in Action Series: Auditing Compliance and ...

This is the third post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...