topic Re: Combine dynamic fields into a multivalued field in a search in Splunk Search

Combine dynamic fields into a multivalued field in a search

shivanshu1593 — Fri, 02 Jul 2021 13:12:17 GMT

Hi All,

I'm working on a search, where I currently have the following:

..base search..

| table static_name, static_time, static_title, static_owner, static_id, static_description

Apart from static_title, static_time, static_id, the other fields are dynamic (they change as the search runs with different inputs. static_owner gets replaced by file_name, other fields like fqdn, process, event_hash etc get added to the search)

What I'm trying to achieve is that the static values, which always are present in the search, should be shown as separate fields, while dynamic fields are merged into a new, multivalued field called Combined_Field, irrespective of the new fields showing up. They should be merged into them.

| table static_time, static_owner, static_id, Combined_Field (a multivalued field, comprising the values of all the dynamic fields)

..base search..

| table static_name, static_time, static_title, static_owner, static_id, static_description

| eval Combined_Field = null()

| foreach * [ eval Combined_Field=if('<<field>>'==static_time OR '<<field>>'==static_owner OR '<<field>>'==static_id, '<<field>>', mvappend('<<field>>', Combined_Field)) ]

| table static_time, static_owner, static_id, Combined_Field

The Combined_Field always remains empty. Could anyone check and let me know as to what am I doing wrong. Or if this can be achieved via a different approach. I've always tried doing the foreach command with case instead of if, no luck.

Thank you in advance,

Re: Combine dynamic fields into a multivalued field in a search

ITWhisperer — Fri, 02 Jul 2021 13:25:55 GMT

You need to check is Combined_Field is null before trying to mvappend to it.

| foreach * [ eval Combined_Field=if('<<field>>'==static_time OR '<<field>>'==static_owner OR '<<field>>'==static_id, '<<field>>', if(isnull(Combined_Field),'<<field>>',mvappend('<<field>>', Combined_Field))) ]

Re: Combine dynamic fields into a multivalued field in a search

shivanshu1593 — Fri, 02 Jul 2021 14:01:58 GMT

Hello @ITWhisperer ,

Thank you for the suggestion. I tried your suggestion, but the field Combined_Fields is still blank.

I used | eval Combined_Fields = null() before foreach. It didn't help either.

Re: Combine dynamic fields into a multivalued field in a search

ITWhisperer — Fri, 02 Jul 2021 14:41:44 GMT

Try the fields in the other order in the mvappend

Re: Combine dynamic fields into a multivalued field in a search

shivanshu1593 — Fri, 02 Jul 2021 14:48:54 GMT

It's still blank. Fields didn't get append to the field Combined_Fields. All are showing up as separate fields in the table command and Combined_Fields is empty.

Re: Combine dynamic fields into a multivalued field in a search

ITWhisperer — Fri, 02 Jul 2021 14:54:07 GMT

Did you put the field names in double quotes - also, you should check if <<field>> is Combined_Field so it doesn't get duplicated.

| foreach * [ eval Combined_Field=if('<<field>>'=="static_time" OR '<<field>>'=="static_owner" OR '<<field>>'=="static_id" OR '<<field>>'=="Combined_Field", '<<field>>', if(isnull(Combined_Field),'<<field>>',mvappend(Combined_Field,'<<field>>'))) ]

Re: Combine dynamic fields into a multivalued field in a search

shivanshu1593 — Fri, 02 Jul 2021 15:07:03 GMT

I did try running the search with field names under double quotes, it still remained blank.

Regarding <<field>> not being Combined_Fields, not really sure how to check that. Could you please point me to a direction, using which I can take a look.

Re: Combine dynamic fields into a multivalued field in a search

ITWhisperer — Fri, 02 Jul 2021 15:23:29 GMT

I had included it already - thinking about it, perhaps you need the <<field>> in double quotes too when checking for the names.

| foreach * [ eval Combined_Field=if("<<field>>"=="static_time" OR "<<field>>"=="static_owner" OR "<<field>>"=="static_id" OR "<<field>>"=="Combined_Field", '<<field>>', if(isnull(Combined_Field),'<<field>>',mvappend(Combined_Field,'<<field>>'))) ]

Re: Combine dynamic fields into a multivalued field in a search

shivanshu1593 — Fri, 02 Jul 2021 15:51:21 GMT

I tried with "<<field>>" as well. Still no results. It's a puzzler. Not really sure what's not working here.

Re: Combine dynamic fields into a multivalued field in a search

ITWhisperer — Fri, 02 Jul 2021 17:34:21 GMT

Try with <<FIELD>>

Re: Combine dynamic fields into a multivalued field in a search

shivanshu1593 — Fri, 02 Jul 2021 17:44:49 GMT

Tried that too. On passing <<field>> without any quotes around it, the if statement fails.

Re: Combine dynamic fields into a multivalued field in a search

ITWhisperer — Fri, 02 Jul 2021 17:46:59 GMT

I meant use FIELD in caps not lowercase

Re: Combine dynamic fields into a multivalued field in a search

shivanshu1593 — Fri, 02 Jul 2021 18:14:45 GMT

My bad. I tried with <<FIELD>> and it has appended all the values, including static_time, static_owner, static_id, which we tried to filter out in the if block.

Re: Combine dynamic fields into a multivalued field in a search

ITWhisperer — Fri, 02 Jul 2021 20:03:37 GMT

So you need caps and double quotes

| foreach * [ eval Combined_Field=if("<<FIELD>>"=="static_time" OR "<<FIELD>>"=="static_owner" OR "<<FIELD>>"=="static_id" OR "<<FIELD>>"=="Combined_Field", '<<FIELD>>', if(isnull(Combined_Field),'<<FIELD>>',mvappend(Combined_Field,'<<FIELD>>'))) ]

Re: Combine dynamic fields into a multivalued field in a search

shivanshu1593 — Fri, 02 Jul 2021 21:54:12 GMT

I've tried to implement that, but its messing up with the fields. There are 17 fields in total in the current iteration, 14 should be in the multivalued field and 3 outside. However, using double quotes and captial fields is reducing the number of fields in the multivalued field down to 8.

Re: Combine dynamic fields into a multivalued field in a search

ITWhisperer — Sat, 03 Jul 2021 07:00:33 GMT

So the process works? Now, it is down to the actual field names. Which field names work, and which ones don't?

Re: Combine dynamic fields into a multivalued field in a search

shivanshu1593 — Mon, 05 Jul 2021 17:01:31 GMT

Hello,

None of the fields are working. All of them are present in the multivalued field, Combined_Fields. On covering them and <<FIELD>> in double quotes, they still land up in the multivalued field.

When I put "<<FIELD>>" == "Combined_Field", then out of 17 fields of this iteration, at least 4 disappear from the multivalued field, leaving only 10 in there instead of 14.