Splunk Search

Concatenating fields in a field transformation

wennebo1
Explorer

We are trying to extract both fields and their names from events that have a variable number of elements. We have determined that using a field transformation is the best way to do this, in order to have multi-valued fields. The names of the fields will be built using the elements in the event.

An example event:

 2019-06-21 14:30 Total Points Player: Red Score: 17,Player: Blue Score: 8,Player: Green Score: 12,Player: Blue Score: 11,Player: Yellow Score: 7,Player: Yellow Score: 10

We would like to extract several fields that look like the this:

 Player_Red_Score: 17
 Player_Blue_Score: 8
                   11
 Player_Green_Score: 12
 Player_Yellow_Score: 7
                     10

The field transformation regular expression is:

 (\w+) Score: (\d+)

And its format is:

 Player_$1_Score::$2

But we end up with fields named Player_$1_Score.

How can we define custom names for these fields by concatenating static text with an extracted field name? I see that this can be done as shown in the documentation here:
https://docs.splunk.com/Documentation/Splunk/7.3.0/Data/Configureindex-timefieldextraction

Any idea what might be going wrong?

0 Karma
1 Solution

wennebo1
Explorer

After gaining a better understanding of search-time and index-time extractions, I found out what the problem was.

The transformations available for editing in Splunk Cloud can only be used for search-time extractions. From the page on field extractions:

You cannot manage index-time field extractions in Splunk Web.

And, from the page on transformations.conf:

* At index time only, you can use FORMAT to create concatenated fields:
* Example: FORMAT = ipaddress::$1.$2.$3.$4

So you will either need to use the method suggested by @DavidHourani, or submit a ticket to Splunk to request a new index-time field extraction.

Not ideal, but this should clarify the problem if anyone else has the same issue.

View solution in original post

0 Karma

wennebo1
Explorer

After gaining a better understanding of search-time and index-time extractions, I found out what the problem was.

The transformations available for editing in Splunk Cloud can only be used for search-time extractions. From the page on field extractions:

You cannot manage index-time field extractions in Splunk Web.

And, from the page on transformations.conf:

* At index time only, you can use FORMAT to create concatenated fields:
* Example: FORMAT = ipaddress::$1.$2.$3.$4

So you will either need to use the method suggested by @DavidHourani, or submit a ticket to Splunk to request a new index-time field extraction.

Not ideal, but this should clarify the problem if anyone else has the same issue.

0 Karma

DavidHourani
Super Champion

Hi @wennebo1,

why not use a regex like this one :

Player\:\sRed Score\:\s+(?<Player_Red_Score>\d+)

You can use the same format for all colors without using an index time extraction:

Player\:\sRed Score\:\s+(?<Player_Red_Score>\d+),Player\:\sBlue\sScore\:\s+(?<Player_Blue_Score>\d+),

Cheers,
David

0 Karma

wennebo1
Explorer

Hello David,

That's a good idea, and might be an option. I would need to make sure I know all the different players in advance, and update if new ones are added, which could get a little annoying.

Even so, I would still need to use a field transformation to allow each field to hold multiple values.

0 Karma

skalliger
SplunkTrust
SplunkTrust

Hey there.

Yes, some ideas. From your URL provided:

In general, you should try to extract your fields at search time. However, there are times when you might find reason to add custom indexed fields.
Do that instead: https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/ExtractfieldsinteractivelywithIFX

I don't see a reason why you'd want to do index-time extractions here. It's just increasing disk space needed and rarely of use.

Skalli

0 Karma

wennebo1
Explorer

Unfortunately, I cannot use the field extractor for two reasons. First, because the resultant fields need to have multiple values per field. Second, because the field extractor cannot use event data in the field names. Neither of these can be done with the field extractor.

0 Karma

anthonymelita
Contributor

Sounds like you're trying to do index time extraction with transforms. You may want to include your actual stanza to get the best advice. I'm not sure but you might need quotations around the captures, "$1" and "$2".

0 Karma

wennebo1
Explorer

I'm using Splunk Cloud, so I cannot edit the .conf files directly. I'm restricted to only the web interface.

I did try putting double quotes around the captures, but it seems they are ignored. Player_"$1"_Score becomes Player_$1_Score. I tried reversing it too, just in case. "Player_"$1"_Score" also becomes Player_$1_Score.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...