Concatenating fields in a field transformation

[wennebo1]

We are trying to extract both fields and their names from events that have a variable number of elements. We have determined that using a field transformation is the best way to do this, in order to have multi-valued fields. The names of the fields will be built using the elements in the event.

An example event:

 2019-06-21 14:30 Total Points Player: Red Score: 17,Player: Blue Score: 8,Player: Green Score: 12,Player: Blue Score: 11,Player: Yellow Score: 7,Player: Yellow Score: 10

We would like to extract several fields that look like the this:

 Player_Red_Score: 17
 Player_Blue_Score: 8
                   11
 Player_Green_Score: 12
 Player_Yellow_Score: 7
                     10

The field transformation regular expression is:

 (\w+) Score: (\d+)

And its format is:

 Player_$1_Score::$2

But we end up with fields named Player_$1_Score.

How can we define custom names for these fields by concatenating static text with an extracted field name? I see that this can be done as shown in the documentation here:
https://docs.splunk.com/Documentation/Splunk/7.3.0/Data/Configureindex-timefieldextraction

Any idea what might be going wrong?

[wennebo1 · Accepted solution]

After gaining a better understanding of search-time and index-time extractions, I found out what the problem was.

The transformations available for editing in Splunk Cloud can only be used for search-time extractions. From the page on field extractions:

You cannot manage index-time field extractions in Splunk Web.

And, from the page on transformations.conf:

* At index time only, you can use FORMAT to create concatenated fields:
* Example: FORMAT = ipaddress::$1.$2.$3.$4

So you will either need to use the method suggested by @DavidHourani, or submit a ticket to Splunk to request a new index-time field extraction.

Not ideal, but this should clarify the problem if anyone else has the same issue.

[DavidHourani]

Hi @wennebo1,

why not use a regex like this one :

Player\:\sRed Score\:\s+(?<Player_Red_Score>\d+)

You can use the same format for all colors without using an index time extraction:

Player\:\sRed Score\:\s+(?<Player_Red_Score>\d+),Player\:\sBlue\sScore\:\s+(?<Player_Blue_Score>\d+),

Cheers,
David

[wennebo1]

Hello David,

That's a good idea, and might be an option. I would need to make sure I know all the different players in advance, and update if new ones are added, which could get a little annoying.

Even so, I would still need to use a field transformation to allow each field to hold multiple values.

[skalliger]

Hey there.

Yes, some ideas. From your URL provided:

In general, you should try to extract your fields at search time. However, there are times when you might find reason to add custom indexed fields.
Do that instead: https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/ExtractfieldsinteractivelywithIFX

I don't see a reason why you'd want to do index-time extractions here. It's just increasing disk space needed and rarely of use.

Skalli

[wennebo1]

Unfortunately, I cannot use the field extractor for two reasons. First, because the resultant fields need to have multiple values per field. Second, because the field extractor cannot use event data in the field names. Neither of these can be done with the field extractor.

[anthonymelita]

Sounds like you're trying to do index time extraction with transforms. You may want to include your actual stanza to get the best advice. I'm not sure but you might need quotations around the captures, "$1" and "$2".

[wennebo1]

I'm using Splunk Cloud, so I cannot edit the .conf files directly. I'm restricted to only the web interface.

I did try putting double quotes around the captures, but it seems they are ignored. Player_"$1"_Score becomes Player_$1_Score. I tried reversing it too, just in case. "Player_"$1"_Score" also becomes Player_$1_Score.