topic Re: removing duplicates in a single field in Splunk Search

How to removine duplicates in a single field?

smanojkumar — Mon, 15 May 2023 13:29:42 GMT

Hi There!

Good day,

I need to remove repeated entries of same values in single field, I'm unable to separate into single values by using values() , mvsplit commands,

Actual one -

src_name serial

item-s1028501

5cd022g2wn 5cd022g2wn 5cd022g2wn 5cd022g2wn 5cd022g2wn 5cd022g2wn 5cd022g2wn 5cd022g2wn

Expected one -

src_name serial

item-s1028501

5cd022g2wn

I just mentioned only one values in src_name, we are having multiple values in src_name to work with,

Thanks in Advance !

Re: removing duplicates in a single field

ITWhisperer — Mon, 15 May 2023 08:15:09 GMT

| eval unique=mvdedup(split(serial," "))

Re: removing duplicates in a single field

TrangCIC81 — Mon, 15 May 2023 08:16:06 GMT

Is the name of the field containing duplicates that you want to remove "src_name"?
if so you can use the below

<your_search> | dedup src_name

Re: removing duplicates in a single field

smanojkumar — Mon, 15 May 2023 08:37:25 GMT

Hi @ITWhisperer ,

After applying the comamand, I got this,

The data is from lookup,

serial src_name length unique

5cd022g2wn 5cd022g2wn 5cd022g2wn 5cd022g2wn 5cd022g2wn 5cd022g2wn 5cd022g2wn

item-s1028501

2718

5cd022g2wn 5cd022g2wn 5cd022g2wn 5cd022g2wn 5cd022g2wn 5cd022g2wn 5cd022g2wn

Re: removing duplicates in a single field

ITWhisperer — Mon, 15 May 2023 08:43:03 GMT

The example you posted has the values separated by spaces (which is what my solution deals with). If your serial field is not actually like that, which would cause the solution to not give the expected answer, you will have to explain what you actual field contains. For example, is it already a multivalue field, or are the values separated by tabs or commas. If you don't provide an accurate representation of your data, you cannot expect an accurate solution to be provided.

Re: removing duplicates in a single field

smanojkumar — Mon, 15 May 2023 08:53:39 GMT

Here is the actual data in JSON format,

{"preview":false,"lastrow":true,"result":{"serial":"5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn\n5cd022g2wn ","src_name":"item-s1028501","length":"2718"}}

Re: removing duplicates in a single field

ITWhisperer — Mon, 15 May 2023 09:28:33 GMT

Since you appear to have a trailing space as well as have \n delimiters, try this

| eval unique=mvdedup(split(trim(serial),"\n"))

Re: removing duplicates in a single field

smanojkumar — Mon, 15 May 2023 09:57:20 GMT

Hi @ITWhisperer ,

I got the same output as previous,

Thanks in Advance!

Re: removing duplicates in a single field

ITWhisperer — Mon, 15 May 2023 10:06:20 GMT

Not sure why the split wasn't recognising the \n - try replacing them with commas, and splitting by those

| eval unique=mvdedup(split(replace(trim(serial_number),"\n",","),","))

Re: removing duplicates in a single field

smanojkumar — Mon, 15 May 2023 10:16:32 GMT

Thanks @ITWhisperer , It worked.