topic Filter out all elements of a list which match the first element in Splunk Search

Filter out all elements of a list which match the first element

BG_Splunk — Fri, 27 Dec 2024 21:35:55 GMT

I'm a bit stumped on this problem. Before I jump into the issue, there's a couple of restrictions:

I'm working in an environment that is running an old version of Splunk which does not have access to the mvmap() function. Working on getting that updated, but until then I still need to try to get a solution to this problem figured out.
This operation is not the only piece of logic I'm trying to accomplish here. Assume there are other unnamed fields which are already in a specifically sorted way which we do not want to disturb.

I'm attempting to filter out all elements of a list which match the first element, leaving only the elements which are not a match. Here is an example which does work:

| makeresults | eval base = split("A75CD,A75AB,A75CD,A75BA,A75DE",",") | eval mv_to_search=mvindex(base,1,mvcount(base)-1) | eval search_value=mvindex(base,0) | eval COMMENT = "ABOVE IS ALL SETUP, BELOW IS ATTEMPTED SOLUTIONS" | eval filtered_mv=mvfilter(!match(base, "A75CD")) | eval var_type = typeof(search_value) | table base, mv_to_search, search_value, filtered_mv, var_type

However, when I attempt to switch it out for something like the following, it does not work:

| makeresults | eval base = split("A75CD,A75AB,A75CD,A75BA,A75DE",",") | eval mv_to_search=mvindex(base,1,mvcount(base)-1) | eval search_value=mvindex(base,0) | eval COMMENT = "ABOVE IS ALL SETUP, BELOW IS ATTEMPTED SOLUTIONS" | eval filtered_mv=mvfilter(!match(base, mvindex(base,0))) | eval var_type = typeof(search_value) | table base, mv_to_search, search_value, filtered_mv, var_type

I have even attempted to solve it using a foreach command, but was also unsuccessful:

| makeresults | eval base = split("A75CD,A75AB,A75CD,A75BA,A75DE",",") | eval mv_to_search=mvindex(base,1,mvcount(base)-1) | eval search_value=mvindex(base,0) | eval COMMENT = "ABOVE IS ALL SETUP, BELOW IS ATTEMPTED SOLUTIONS" | foreach mode=multivalue base [eval filtered_mv = if('<<ITEM>>'!=mvindex(base,0), mvappend(filtered_mv,'<<ITEM>>'), filtered_mv)] | eval var_type = typeof(search_value) | table base, mv_to_search, search_value, filtered_mv, var_type

I'm open to any other ideas which might accomplish this better or more efficiently. Not sure where I'm going wrong with this one, or whether this idea is even possible.

Re: Filter out all elements of a list which match the first element

bowesmana — Sat, 28 Dec 2024 01:20:44 GMT

@BG_Splunk I you don't have mvmap, you won't have foreach mode=multivalue, but you can use foreach like this

| foreach 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 [ | eval e=mvindex(base, <<FIELD>>), filtered_mv=mvappend(filtered_mv, if(!match(e, search_value), e, null())) ] | eval var_type = typeof(search_value) | table base, mv_to_search, search_value, filtered_mv, var_type

where you just give incrementing numbers which are templated as <<FIELD>> so you can mvindex using that.

mvfilter can't handle more than one field, so the mvindex(base, 0) won't work inside the filter expression.

I'm still using 7.3 in one environment so the above works in that and I used to use that technique before mvmap came along. It does require you to know the max size of the list in advance, but it doesn't have limits I have come up against.

It may also be possible to collapse the MV to a single value and then use some kind of rex/replace to get the matches out, but I've not tried that.

Re: Filter out all elements of a list which match the first element

bowesmana — Sat, 28 Dec 2024 01:24:57 GMT

If this is practical, you can do it with replace, i.e.

| eval elements = mvjoin(mv_to_search, "####") | eval non_matches = replace(elements, search_value, ""), non_matches=split(non_matches, "####"), non_matches=mvfilter(isnotnull(non_matches))

which joins the elements with a known string, gets rid of all the matches, then splits again and removes nulls.

Re: Filter out all elements of a list which match the first element

BG_Splunk — Sat, 28 Dec 2024 01:37:13 GMT

My guy, that's a super smart solution! Thank you very much, I just tried this out and it works beautifully. 🙂

I'm going to have to keep this kind of approach in mind as I go forward with this project. Very creative thinking!

Re: Filter out all elements of a list which match the first element

BG_Splunk — Sat, 28 Dec 2024 01:39:00 GMT

That's a very smart way to do it! I'm going to need some time to dissect how that works, but I know off the bat that the biggest problem is going to be that the max size of the list is going to be variable each time this runs. Still, I think this is super clever and will keep this in mind. 🙂

Re: Filter out all elements of a list which match the first element

bowesmana — Sat, 28 Dec 2024 20:53:13 GMT

If you always know what the upper max list size will be then you can put foreach numbers 0 1...999999 if you really needed as nothing will happen for those outside the actual size of the MV.

If you're doing this in a dashboard, you could technically create a token with the numbered steps and use the token in the foreach, e.g.

| foreach $steps_to_iterate$

where steps to iterate is calculated in a post process search of the list and simply

| stats max(eval(mvcount(list))) as max_list | eval r=mvjoin(mvrange(1, max_list + 1, 1), " ")

with this <done> clause in the dashboard search

<done> <set token="steps_to_iterate">$result.max_list$</set> </done>