topic Re: Default value for transform in Getting Data In

Default value for transform

wollinet — Thu, 15 Jul 2010 23:29:42 GMT

FORMAT = <string>
* The special identifier $0 represents what was in the DEST_KEY before this regex was performed.

Can I use $0 in DEFAULT_VALUE, too ? My intention is to add a default value to the current content of the source key, if the regex doesn't match.

Any help appreciated.

Here's some more information about what I want to do. I need to build the index name from several fields in the record (e.g. stage and logtype). In some cases I can directly use the value from the record in some other cases not. I tried the following:

props.conf:
TRANSFORMS-index_stage = index_stage_prod, index_stage_test, index_logtype
...

transforms.conf:

[index_stage_prod] 
DEST_KEY = _MetaData:Index 
REGEX = (?i)stage=(PROD|BCP).* 
FORMAT = "idx_prod"

[index_stage_test] 
DEST_KEY = _MetaData:Index 
REGEX = (?i)stage=(DEV|TEST).* 
FORMAT = "idx_test"

[index_logtype] 
DEST_KEY = _MetaData:Index 
REGEX = (?i)logtype=([^\s\t\r]*).* 
FORMAT = $0_$1 
DEFAULT_VALUE = $0_unclassified

The seem to be at least two problems: 1) It seems that building the index name incrementally doesn't work. My tests showed that if the first transform was successful the other transforms do not fire. 2) I need "$0" to work in DEFAULT_VALUE

I workaround would be to match all values with one regular expression, but that has some limitations. I will also open a case about that problem.

Re: Default value for transform

Lowell — Fri, 16 Jul 2010 20:53:14 GMT

I don't think this is possible. In the common field extraction (fields loaded at search-time) it is not possible to augment the regex extracted values. So it seems unlikely that you could augment the previous value of the DEST_KEY ($0) with some additional static text that would work when you don't have a match. Certainly there would be ways of accomplishing this using an eval statement, but I don't think you can do this simply with a transformer.

Here is an example demonstrating what I mean about adding static text to a field. This example will not work:

[my-bogus-transformer]
REGEX = \s(\d+)\s
FORMAT = my_field::"$1 (int)"

This does not work, because the text " (int)" is NOT part of the raw text and you can't just arbitrarily add text to extracted fields like this. (Unless you use do index-time field extractions, which I don't recommend here.) My understanding is that this has to do with search performance and the fact that a search like field1=joe is turned into the search joe AND field1="joe".

Re: Default value for transform

wollinet — Mon, 28 Sep 2020 09:15:13 GMT

$0 is used differently here. The documentation is correct as $0 is original value of DEST_KEY. I've already tested that.
What I want is to add something to DEST_KEY (which already contains a value) if the regex matches and a default value if not. The problem is that the default value should be added to DEST_KEY and not overwrite it.

Re: Default value for transform

Lowell — Mon, 19 Jul 2010 22:12:09 GMT

Wollinet, you are correct. I've updated by answer and removed my incorrect understanding of how splunk is using $0 in this case vs the traditional regex meaning of $0. I think an eval based approach is your best (and possibly only) option.

Re: Default value for transform

wollinet — Tue, 20 Jul 2010 20:27:01 GMT

"eval" doesn't help, since I need that functionality during indexing. I want to dynamically choose the index where the record is stored.

Re: Default value for transform

Lowell — Tue, 20 Jul 2010 20:56:29 GMT

Hmm, that's a very different scenario that I thought you were asking about. Index-time transforms don't have the limitation that I described above, but I'm not sure knowing that solves anything.... I think if you update your questions with a use-case example or two, there may be another possible solution. It's also possible this should be a feature request, but either way you'll need to give more details about what your use-case is.

Re: Default value for transform

wollinet — Thu, 29 Jul 2010 20:16:54 GMT

After a lot of testing I'm now sure that $0 doesn't work in DEFAULT_VALUE. I'm gonna file an enhancement request.