Default value for transform

[wollinet]

FORMAT = <string>
* The special identifier $0 represents what was in the DEST_KEY before this regex was performed.

Can I use $0 in DEFAULT_VALUE, too ? My intention is to add a default value to the current content of the source key, if the regex doesn't match.

Any help appreciated.

---

Here's some more information about what I want to do. I need to build the index name from several fields in the record (e.g. stage and logtype). In some cases I can directly use the value from the record in some other cases not. I tried the following:

props.conf:
TRANSFORMS-index_stage = index_stage_prod, index_stage_test, index_logtype
...

transforms.conf:

[index_stage_prod] 
DEST_KEY = _MetaData:Index 
REGEX = (?i)stage=(PROD|BCP).* 
FORMAT = "idx_prod"

[index_stage_test] 
DEST_KEY = _MetaData:Index 
REGEX = (?i)stage=(DEV|TEST).* 
FORMAT = "idx_test"

[index_logtype] 
DEST_KEY = _MetaData:Index 
REGEX = (?i)logtype=([^\s\t\r]*).* 
FORMAT = $0_$1 
DEFAULT_VALUE = $0_unclassified 

The seem to be at least two problems: 1) It seems that building the index name incrementally doesn't work. My tests showed that if the first transform was successful the other transforms do not fire. 2) I need "$0" to work in DEFAULT_VALUE

I workaround would be to match all values with one regular expression, but that has some limitations. I will also open a case about that problem.

[wollinet · Accepted solution]

After a lot of testing I'm now sure that $0 doesn't work in DEFAULT_VALUE. I'm gonna file an enhancement request.

[Lowell]

I don't think this is possible. In the common field extraction (fields loaded at search-time) it is not possible to augment the regex extracted values. So it seems unlikely that you could augment the previous value of the DEST_KEY ($0) with some additional static text that would work when you don't have a match. Certainly there would be ways of accomplishing this using an eval statement, but I don't think you can do this simply with a transformer.

Here is an example demonstrating what I mean about adding static text to a field. This example will not work:

[my-bogus-transformer]
REGEX = \s(\d+)\s
FORMAT = my_field::"$1 (int)"

This does not work, because the text " (int)" is NOT part of the raw text and you can't just arbitrarily add text to extracted fields like this. (Unless you use do index-time field extractions, which I don't recommend here.) My understanding is that this has to do with search performance and the fact that a search like field1=joe is turned into the search joe AND field1="joe".

[Lowell]

Hmm, that's a very different scenario that I thought you were asking about. Index-time transforms don't have the limitation that I described above, but I'm not sure knowing that solves anything.... I think if you update your questions with a use-case example or two, there may be another possible solution. It's also possible this should be a feature request, but either way you'll need to give more details about what your use-case is.

[wollinet]

"eval" doesn't help, since I need that functionality during indexing. I want to dynamically choose the index where the record is stored.

[Lowell]

Wollinet, you are correct. I've updated by answer and removed my incorrect understanding of how splunk is using $0 in this case vs the traditional regex meaning of $0. I think an eval based approach is your best (and possibly only) option.

[wollinet]

$0 is used differently here. The documentation is correct as $0 is original value of DEST_KEY. I've already tested that.
What I want is to add something to DEST_KEY (which already contains a value) if the regex matches and a default value if not. The problem is that the default value should be added to DEST_KEY and not overwrite it.