topic Re: remove repetitive string in string in Splunk Search

remove repetitive string in string

exocore123 — Tue, 27 Jun 2017 20:50:19 GMT

manipulating strings, I had a post before regarding an array, but say I have a field that has value string1+string2+string1, how can I display this value as packaged(string1+string2)

EDIT:
I have a particular field in my logs that is displayed as field="string1+string2+string1" and when I use stats on this field to make a table, however, the string length can be very repetitive field="string1+string1+string1+string1..." or field="string1+string2+string2", so I want to be able to basically make any string with logically more than 1 value to be displayed as package(..) where the .. is the unique strings, so for field="string1+string1" can be packaged(string1) so does field="string1+string1+string1" and so forth, for
field="string1+string2+string2" or field="string1+string2" can be packaged(string1+string2)
Say each string represents different codes that are vastly different from one and another, so wildcard isnt an option.

Re: remove repetitive string in string

niketn — Tue, 27 Jun 2017 21:34:46 GMT

@exocore123, you might have to add sample values(if actual value is not possible please add anonymized string) for us to help you better.

Re: remove repetitive string in string

exocore123 — Tue, 27 Jun 2017 21:40:51 GMT

@niketnilay idk if that helps but i tried to be more specific

Re: remove repetitive string in string

niketn — Tue, 27 Jun 2017 22:10:10 GMT

@exocore123, by values I meant some sample data.

Does your strings in the field actually have plus sign (+) or any other delimiter between them?
Does your field actually have string1 or string2 in it? Or is this sample?

Or is your field value something like following?
field= "abcdefabc"
Where abc is string1 and def is string2.

Re: remove repetitive string in string

exocore123 — Tue, 27 Jun 2017 22:15:31 GMT

The things I put are the literal sample data. The + sign is part of the string. It's literally field="asdw+wkas+asdw", not sure if that clarifies?

Re: remove repetitive string in string

DalJeanis — Tue, 27 Jun 2017 23:05:15 GMT

Try this -

| makeresults | eval myfield="asdw+wkas+asdw" 
| rename COMMENT as "The above creates test data"

| makemv delim="+" myfield 
| eval myfield=if(mvcount(myfield)<=1,myfield,"package(".mvjoin(mvdedup(myfield),"+").")")

with resulting output...

myfield
package(asdw+wkas)

Now, since you are not retaining the info on how many of each were originally there, and since the field already had multiple values in it, you could forego the words "package()" around the altered field if you wanted.

| makemv delim="+" myfield 
| eval myfield=if(mvcount(myfield)<=1,myfield,mvjoin(mvdedup(myfield),"+"))

Re: remove repetitive string in string

exocore123 — Tue, 27 Jun 2017 23:49:08 GMT

Works perfectly, one comment, say I have a string3 I do not want to add after the makemv, say string1+string2+string3, so after makemv it becomes string1 string2 string3, say I want to mvjoin only string1 and string2, is there a selective way to do this?

Re: remove repetitive string in string

DalJeanis — Wed, 28 Jun 2017 01:42:40 GMT

There are two easy ways, depending on how many of string3 you have.

You can use mvfilter ...

| makemv delim="+" myfield 
| eval myfield=mvfilter(NOT match(myfield,"idonotwantthis|orthis"))
| eval myfield=if(mvcount(myfield)=1, myfield, "package(".mvjoin(mvdedup(myfield),"+").")")

If the list of wanted values is shorter than the list of unwanted ones, then remove the NOT and put your desired strings connected by the vertical OR bars (|).

| eval myfield=mvfilter(match(myfield,"iwantthis|andthis|andthese"))

Or you can also use rex in sed mode ...

| makemv delim="+" myfield 
| eval myfield=if(mvcount(myfield)=1, myfield, "package(".mvjoin(mvdedup(myfield),"+").")")

| rename COMMENT as "The first sed gets rid of the undesired strings" 
| rex field=myfield mode=sed "s/idonotwantthis|orthis//g"

| rename COMMENT as  "The second sed gets rid of excess plusses at the front, middle or end of the string."
| rex field=myfield mode=sed "^\+|\+(?=\+)|\+$"

There are also some more complex ways if you have a bunch of these undesired strings to check for.

Re: remove repetitive string in string

exocore123 — Wed, 28 Jun 2017 03:16:05 GMT

I think I get the idea, thanks!