FORMAT = <string>
* The special identifier $0 represents what was in the DEST_KEY before this regex was performed.
Can I use $0 in DEFAULT_VALUE, too ? My intention is to add a default value to the current content of the source key, if the regex doesn't match.
Any help appreciated.
Here's some more information about what I want to do. I need to build the index name from several fields in the record (e.g. stage and logtype). In some cases I can directly use the value from the record in some other cases not. I tried the following:
props.conf:
TRANSFORMS-index_stage = index_stage_prod, index_stage_test, index_logtype
...
transforms.conf:
[index_stage_prod]
DEST_KEY = _MetaData:Index
REGEX = (?i)stage=(PROD|BCP).*
FORMAT = "idx_prod"
[index_stage_test]
DEST_KEY = _MetaData:Index
REGEX = (?i)stage=(DEV|TEST).*
FORMAT = "idx_test"
[index_logtype]
DEST_KEY = _MetaData:Index
REGEX = (?i)logtype=([^\s\t\r]*).*
FORMAT = $0_$1
DEFAULT_VALUE = $0_unclassified
The seem to be at least two problems: 1) It seems that building the index name incrementally doesn't work. My tests showed that if the first transform was successful the other transforms do not fire. 2) I need "$0" to work in DEFAULT_VALUE
I workaround would be to match all values with one regular expression, but that has some limitations. I will also open a case about that problem.
After a lot of testing I'm now sure that $0 doesn't work in DEFAULT_VALUE. I'm gonna file an enhancement request.
After a lot of testing I'm now sure that $0 doesn't work in DEFAULT_VALUE. I'm gonna file an enhancement request.
I don't think this is possible. In the common field extraction (fields loaded at search-time) it is not possible to augment the regex extracted values. So it seems unlikely that you could augment the previous value of the DEST_KEY
($0
) with some additional static text that would work when you don't have a match. Certainly there would be ways of accomplishing this using an eval
statement, but I don't think you can do this simply with a transformer.
Here is an example demonstrating what I mean about adding static text to a field. This example will not work:
[my-bogus-transformer]
REGEX = \s(\d+)\s
FORMAT = my_field::"$1 (int)"
This does not work, because the text " (int)" is NOT part of the raw text and you can't just arbitrarily add text to extracted fields like this. (Unless you use do index-time field extractions, which I don't recommend here.) My understanding is that this has to do with search performance and the fact that a search like field1=joe
is turned into the search joe AND field1="joe"
.
Hmm, that's a very different scenario that I thought you were asking about. Index-time transforms don't have the limitation that I described above, but I'm not sure knowing that solves anything.... I think if you update your questions with a use-case example or two, there may be another possible solution. It's also possible this should be a feature request, but either way you'll need to give more details about what your use-case is.
"eval" doesn't help, since I need that functionality during indexing. I want to dynamically choose the index where the record is stored.
Wollinet, you are correct. I've updated by answer and removed my incorrect understanding of how splunk is using $0
in this case vs the traditional regex meaning of $0
. I think an eval
based approach is your best (and possibly only) option.
$0 is used differently here. The documentation is correct as $0 is original value of DEST_KEY. I've already tested that.
What I want is to add something to DEST_KEY (which already contains a value) if the regex matches and a default value if not. The problem is that the default value should be added to DEST_KEY and not overwrite it.