topic Create and De-reference New Field Names in Splunk Search

Create and De-reference New Field Names

tom_porter — Fri, 08 Dec 2023 18:59:23 GMT

I have Linux audit records that have a field called type and fields with the naming convention lower(type).field. I want to be able to combine type, as a prefix, and a set of suffixes to create new field names that exist in the data. For example, I have a type called FILE_WATCH and fields called file_watch.exe, file_watch.egid, file_watch.comm, etc.

I want to develop a dashboard table by type and suffix in Splunk to show if a particular field exists for a type. So going back to my example using type=FILE_WATCH, how can I create a new field name along these lines?

base = lower(type)
exe={base}.".exe" # does not work, but you get the idea.

with exe now equal to the field name, I want to be able to de-reference the new field name to see if it exists.

Re: Create and De-reference New Field Names

yuanliu — Fri, 08 Dec 2023 19:34:23 GMT

I'm not sure if I get the idea. Do you mean to use this?

base = lower(type) exe=base.".exe"

What is the intention of the curly brackets? (No, {base} is not a valid expression.)

Re: Create and De-reference New Field Names

richgalloway — Fri, 08 Dec 2023 20:16:07 GMT

Curly braces on the LHS of an eval says to use the contents of the field as the new field name.

See if this example query helps explain

| makeresults | eval base="file_watch" | eval {base}= base.".exe" | eval {base}exe=base.".exe"

Re: Create and De-reference New Field Names

yuanliu — Fri, 08 Dec 2023 21:53:29 GMT

I stand corrected. Thanks, @richgalloway !

Now, @tom_porter will need to explain what "does not work" means. To be clear, this phrase should be forbidden in a forum like this as it conveys very little information in the best scenarios.

Illustrate/mock your data (anonymize as needed),
show code you tried,
show actual results (anonymize as needed),
illustrate/mock desired results,
explain the logic connecting data and desired results if it is not painfully obvious.

Also explain the difference between actual results and desired results if it is not painfully obvious.

Re: Create and De-reference New Field Names

tom_porter — Tue, 12 Dec 2023 12:15:46 GMT

I was trying something along the lines of dynamic field creation. At issue is that we have multiple dot notation field names with different prefixes, but a common suffix. (e.g.: file_watch.sgid and execve.sgid). There are about 40 prefixes and 50 or more suffixes. Not all prefixes have all suffixes. What I wanted to do was to create a dashboard that would show the prefixes as rows, and the suffixes as columns, with x marking cells with non-null values for prefix.suffix based on a search over the last 24 hours.

Re: Create and De-reference New Field Names

dtburrows3 — Wed, 13 Dec 2023 00:09:32 GMT

Not sure if I am interpreting your question correctly but I gave it a shot.

So given that the are many different fieldnames with dot notation.

You are trying to get a final table of something like this?

I was able to achieve this by utilizing a foreach loop

| makeresults | eval "tmp.exe"="value1" | append [ | makeresults | eval "noop.spl"="value2" ] | append [ | makeresults | eval "tmp.spl"="value3" ] | append [ | makeresults | eval "foo.exe"="value4" ] | append [ | makeresults | eval "tmp.tgz"="value5" ] | append [ | makeresults | eval "foo.tgz"="value6", "tmp.exe"="value7" ] ``` Gather unique fieldnames as values of a new field ``` | foreach *.* [ | eval existing_fieldname=if( isnotnull('<<FIELD>>'), mvappend( 'existing_fieldname', "<<FIELD>>" ), 'existing_fieldname' ) ] ``` Parse out prefix and suffix of the new field ``` | eval prefix=case( isnull(existing_fieldname), null(), mvcount(existing_fieldname)==1, mvindex(split(existing_fieldname, "."), 0), mvcount(existing_fieldname)>1, mvmap(existing_fieldname, mvindex(split(existing_fieldname, "."), 0)) ), suffix=case( isnull(existing_fieldname), null(), mvcount(existing_fieldname)==1, mvindex(split(existing_fieldname, "."), 1), mvcount(existing_fieldname)>1, mvmap(existing_fieldname, mvindex(split(existing_fieldname, "."), 1)) ) ``` Use chart function to display unique combos of prefix/suffix from inherited fieldnames ``` | chart limit=50 count as count over prefix by suffix ``` Replace numbers in the table with "X" to signify that the prefix/suffix combo was found in the data ``` | foreach * [ | eval <<FIELD>>=if( NOT "<<FIELD>>"=="prefix", if( '<<FIELD>>'>0, "X", null() ), '<<FIELD>>' ) ]

Re: Create and De-reference New Field Names

tom_porter — Wed, 13 Dec 2023 18:59:14 GMT

Yes!!! That second table. Thank you....will try out your solution later this week. Much appreciation to you all for chiming in on this.

Re: Create and De-reference New Field Names

tscroggins — Sat, 06 Jan 2024 17:35:36 GMT

Jumping in on an aging topic, but you may be able to simplify the SPL, albeit with an unknown impact to performance. (Always test!)

| makeresults format=json data="[{\"foo\": {\"field1\": \"value1\", \"field2\": \"value2\"}}, {\"bar\": {\"field1\": \"value3\", \"field2\": \"value4\"}}, {\"baz\": {\"field2\": \"value5\", \"field3\": \"value6\"}}]" | spath ``` end test data ``` | table *.* | transpose | rex field=column "(?<prefix>[^.]+)\\.(?<suffix>.+)" | foreach row* [ eval value=coalesce('<<FIELD>>', value) ] | xyseries prefix suffix value