Getting Data In

Default value for transform

wollinet
Path Finder
FORMAT = <string>
* The special identifier $0 represents what was in the DEST_KEY before this regex was performed.

Can I use $0 in DEFAULT_VALUE, too ? My intention is to add a default value to the current content of the source key, if the regex doesn't match.

Any help appreciated.


Here's some more information about what I want to do. I need to build the index name from several fields in the record (e.g. stage and logtype). In some cases I can directly use the value from the record in some other cases not. I tried the following:

props.conf:
TRANSFORMS-index_stage = index_stage_prod, index_stage_test, index_logtype
...

transforms.conf:

[index_stage_prod] 
DEST_KEY = _MetaData:Index 
REGEX = (?i)stage=(PROD|BCP).* 
FORMAT = "idx_prod"

[index_stage_test] 
DEST_KEY = _MetaData:Index 
REGEX = (?i)stage=(DEV|TEST).* 
FORMAT = "idx_test"

[index_logtype] 
DEST_KEY = _MetaData:Index 
REGEX = (?i)logtype=([^\s\t\r]*).* 
FORMAT = $0_$1 
DEFAULT_VALUE = $0_unclassified 

The seem to be at least two problems: 1) It seems that building the index name incrementally doesn't work. My tests showed that if the first transform was successful the other transforms do not fire. 2) I need "$0" to work in DEFAULT_VALUE

I workaround would be to match all values with one regular expression, but that has some limitations. I will also open a case about that problem.

0 Karma
1 Solution

wollinet
Path Finder

After a lot of testing I'm now sure that $0 doesn't work in DEFAULT_VALUE. I'm gonna file an enhancement request.

View solution in original post

0 Karma

wollinet
Path Finder

After a lot of testing I'm now sure that $0 doesn't work in DEFAULT_VALUE. I'm gonna file an enhancement request.

0 Karma

Lowell
Super Champion

I don't think this is possible. In the common field extraction (fields loaded at search-time) it is not possible to augment the regex extracted values. So it seems unlikely that you could augment the previous value of the DEST_KEY ($0) with some additional static text that would work when you don't have a match. Certainly there would be ways of accomplishing this using an eval statement, but I don't think you can do this simply with a transformer.

Here is an example demonstrating what I mean about adding static text to a field. This example will not work:

[my-bogus-transformer]
REGEX = \s(\d+)\s
FORMAT = my_field::"$1 (int)"

This does not work, because the text " (int)" is NOT part of the raw text and you can't just arbitrarily add text to extracted fields like this. (Unless you use do index-time field extractions, which I don't recommend here.) My understanding is that this has to do with search performance and the fact that a search like field1=joe is turned into the search joe AND field1="joe".

Lowell
Super Champion

Hmm, that's a very different scenario that I thought you were asking about. Index-time transforms don't have the limitation that I described above, but I'm not sure knowing that solves anything.... I think if you update your questions with a use-case example or two, there may be another possible solution. It's also possible this should be a feature request, but either way you'll need to give more details about what your use-case is.

0 Karma

wollinet
Path Finder

"eval" doesn't help, since I need that functionality during indexing. I want to dynamically choose the index where the record is stored.

0 Karma

Lowell
Super Champion

Wollinet, you are correct. I've updated by answer and removed my incorrect understanding of how splunk is using $0 in this case vs the traditional regex meaning of $0. I think an eval based approach is your best (and possibly only) option.

0 Karma

wollinet
Path Finder

$0 is used differently here. The documentation is correct as $0 is original value of DEST_KEY. I've already tested that.
What I want is to add something to DEST_KEY (which already contains a value) if the regex matches and a default value if not. The problem is that the default value should be added to DEST_KEY and not overwrite it.

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...