topic Is it possible to remove values specified in a field from a list of values in another field? in Splunk Search

Is it possible to remove values specified in a field from a list of values in another field?

andrewtrobec — Thu, 28 Dec 2017 17:39:30 GMT

Hello,

Given two list fields add and remove, as well as a currentList list field, is it possible to create a new fieldupdatedListso that gets updated with add values and without remove values?

add,remove,currentList,updatedList
value1,value2,value2 value3 value4, value1 value3 value4
value5,value3 value4,value1 value3 value4,value1 value5

Any hints would be great!

Best regards,

Andrew

Re: Is it possible to remove values specified in a field from a list of values in another field?

somesoni2 — Thu, 28 Dec 2017 19:22:09 GMT

Is field currentList multivalued field? Also, are add and remove multivalued field or single valued?

Re: Is it possible to remove values specified in a field from a list of values in another field?

andrewtrobec — Thu, 28 Dec 2017 19:34:42 GMT

currentList, add, and remove are all multivalue fields. I've figured the add part out:

| eval currentList=mvdedup(mvappend(currentList, add))

I just need to figure out the remove part.

Re: Is it possible to remove values specified in a field from a list of values in another field?

somesoni2 — Thu, 28 Dec 2017 20:09:02 GMT

Unfortunately, there is no easy/direct way for removal from multivalued field. One method could be like this.

your current search with fields add, remove and currentList
| eval currentList=mvdedup(mvappend(currentList, add))
| nomv remove 
| streamstats count as sno
| mvexpand currentList
| where len(replace(remove,currentList,"")=len(remove)
| stats values(*) as * by sno | fields - sno

Re: Is it possible to remove values specified in a field from a list of values in another field?

andrewtrobec — Thu, 28 Dec 2017 20:42:57 GMT

I've cut and pasted the code into my search but it doesn't seem to work, it just kinda messes up the fields. I'll deconstruct the logic and play around with it. Thanks!

Re: Is it possible to remove values specified in a field from a list of values in another field?

micahkemp — Thu, 28 Dec 2017 21:05:57 GMT

Here's a run-anywhere possibility:

| makeresults | eval add="value1", remove="value2", current="value2 value3 value4"
| append [| makeresults | eval add="value5", remove="value3 value4", current="value1 value3 value4"]
| makemv add 
| makemv remove 
| makemv current
| eval current_plus_add=mvappend(add, current)
| streamstats window=1 current=t values(current_plus_add) AS distinct_current_plus_add, values(remove) AS distinct_remove
| eval distinct_current_plus_add_plus_distinct_remove=mvappend(distinct_current_plus_add, distinct_remove)
| streamstats count AS serial
| stats count BY distinct_current_plus_add_plus_distinct_remove serial
| search count=1
| stats list(distinct_current_plus_add_plus_distinct_remove) AS after_add_remove BY serial

Re: Is it possible to remove values specified in a field from a list of values in another field?

andrewtrobec — Thu, 28 Dec 2017 23:14:19 GMT

Beautiful! I hope that a future feature for multivalues will be mvremove or something like that so we can easily remove values instead of tricky workarounds!

Re: Is it possible to remove values specified in a field from a list of values in another field?

niketn — Fri, 29 Dec 2017 02:46:21 GMT

@andrewtrobec, you can create your own custom commands 🙂

https://docs.splunk.com/Documentation/Splunk/latest/Search/Customsearchcommandshape
http://dev.splunk.com/view/python-sdk/SP-CAAAEU2