topic Re: Remove Alike Values in a Field in Splunk Search

Remove Alike Values in a Field

whitefang1726 — Wed, 05 Jan 2022 09:43:18 GMT

Hello Splunk Answers, How can I remove this duplicate line? See sample below:

From:

row1 row2 row3
1.1.1.1 XXX alpha.splunk.com
alpha
2.2.2.2 YYY beta.splunk.com
BETA
3.3.3.3 ZZZ delta.splunkanswers.com
delta
4.4.4.4 AAA abcdefgh

to:
row1 row2 row3
1.1.1.1 XXX alpha.splunk.com
2.2.2.2 YYY beta.splunk.com
3.3.3.3 ZZZ delta.splunkanswers.com
4.4.4.4 AAA abcdefgh

Thanks!

Re: Remove Alike Values in a Field

ITWhisperer — Wed, 05 Jan 2022 10:05:03 GMT

It is not clear what these "lines" are - are they all in a single field called row3 with linebreaks? are they multivalues? have they been created by stats values/list functions, or transaction commands?

Re: Remove Alike Values in a Field

whitefang1726 — Wed, 05 Jan 2022 12:38:33 GMT

Okay, here's another example.

from:
field1 field2 field3 field4
row1 1.1.1.1 XXX alpha.splunk.com
alpha
row2 2.2.2.2 YYY beta.splunk.com
BETA
row3 3.3.3.3 ZZZ delta.splunkanswers.com
delta
row4 4.4.4.4 AAA abcdefgh

to:
field1 field2 field3 field4
row1 1.1.1.1 XXX alpha.splunk.com
row2 2.2.2.2 YYY beta.splunk.com
row3 3.3.3.3 ZZZ delta.splunkanswers.com
row4 4.4.4.4 AAA abcdefgh

or:

field2 field3 field4
1.1.1.1 XXX alpha.splunk.com
2.2.2.2 YYY beta.splunk.com
3.3.3.3 ZZZ delta.splunkanswers.com
4.4.4.4 AAA abcdefgh

Re: Remove Alike Values in a Field

ITWhisperer — Wed, 05 Jan 2022 12:42:15 GMT

Sorry, that doesn't really help. Can you provide (sanitised) examples of your raw events and the searches you are currently using to get your current results?

Re: Remove Alike Values in a Field

whitefang1726 — Thu, 06 Jan 2022 05:40:16 GMT

This is a sample output using stats values. Sorry, I give a wrong example. Here's the current one.

from:
field1 field2 field3 field4
row1 1.1.1.1 XXX alpha.splunk.com
alpha
row2 2.2.2.2 YYY beta.splunk.com
BETA
row3 3.3.3.3 ZZZ delta.splunkanswers.com
delta
row4 4.4.4.4 AAA abcdefgh

To:
field1 field2 field3 field4
row1 1.1.1.1 XXX alpha.splunk.com
row2 2.2.2.2 YYY beta.splunk.com
row3 3.3.3.3 ZZZ delta.splunkanswers.com
row4 4.4.4.4 AAA abcdefgh

So, for field4, alpha.splunk.com and alpha is just one value (multi-value), beta and beta.splunk.com is one value (multi-value), so on...
I want to remove the redundant names (hostname only and host with FQDN) as long it same value (example: alpha and alpha.splunk.com)

Re: Remove Alike Values in a Field

johnhuang — Thu, 06 Jan 2022 06:00:46 GMT

If they are multivalue fields and you want the first:

| eval row3=mvindex(row3, 0)

Re: Remove Alike Values in a Field

whitefang1726 — Fri, 07 Jan 2022 07:59:24 GMT

This works, thanks!