regex transformation not working in transforms.conf where it does at search time?

[jmartens]

I have the following data in a key (called test_key through a field extraction) I want to split:

domain\firstname.lastname|38372|VENDOR_CODE

I try to extract some values and assign them to Application and ProcessId key from another field extraction. At search time the extractions seem to work with the following:

| rex field=test_key "\|(?<ProcessId>\d+)\|" | rex field=test_key "\|(?<Application>\D+)$"

However when I define them in my transforms.conf (currently local/transforms.conf for testing) in the app I am bundling everything in I can not get them to work, this is the output as defined while entering it through the web interface:

[Application]
CLEAN_KEYS = 0
REGEX = \|(?<Application>\D+)$
SOURCE_KEY = test_key

[ProcessId]
CLEAN_KEYS = 0
REGEX = "\|(?<ProcessId>\d+)\|"
SOURCE_KEY = test_key

At search time I do not see the fields Application and ProcessId appearing, where I do see them as soon as I add the regex stanza to the search at search time. Any clues on how to get my transformations working?

[somesoni2]

How about you merge the extraction with your current EXTRACT entry , like this

EXTRACT-test_format= (?:[^\t\n]*\t){2}(?P<facility>[^\t]+)\t(?P<loglevel>[^\t]+)\t(?P<time2>[^\t]+)\t(?P<raw_serialno>[^\t]+)\t(?<test_key>([^\|]+\|(?<ProcessId>\d+)\|(?<Application>\w+)))[^\t\n]*\t(?P<unknown>[^\t]+)\t(?P<message>.+)

[xpac]

Can you please show your props.conf? Depending on that, it might be possible that test_key is extracted AFTER your new extractions - therefore it doesn't work.

[jmartens]

I doubt that, the node field is in the first extraction in my apps props.conf field.

[test_format]
category = Custom
description = 
disabled = false
pulldown_type = true

EXTRACT-test_format= (?:[^\t\n]*\t){2}(?P<facility>[^\t]+)\t(?P<loglevel>[^\t]+)\t(?P<time2>[^\t]+)\t(?P<raw_serialno>[^\t]+)\t(?P<test_key>[^\t]+)[^\t\n]*\t(?P<unknown>[^\t]+)\t(?P<message>.+)
...

[FrankVl]

Do you have REPORT references in your props.conf to your transforms.conf stanzas? Transforms.conf stuff doesn't work on its own 🙂

[xpac]

That would definetely be something to check. 🙂

Also, EXTRACT happens before REPORT, but they're sorted by ASCII-order, not by the order they appear in the file.
Therefore (just as an example), an EXTRACT-test_format would happen after an EXTRACT-application, and it would also happen after an EXTRACT-ZZZ, because uppercase letter come before lowercase in ASCII. Just want to make sure.
For more infos on search-time sequence order, check this:
http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Searchtimeoperationssequence