Starting with the data in an event:
Lines in Single Event:
PosTransactionProperties.PosTransactionPropertyCode=RECEIPT_EMAIL_ADDRESS PosTransactionProperties.PosTransactionPropertyValuefirstname.lastname@example.org PosTransactionProperties.PosTransactionPropertyCode=RECEIPT_DELIVERY_METHOD PosTransactionProperties.PosTransactionPropertyValue=EMAIL+PAPER PosTransactionProperties.PosTransactionPropertyCode=CUSTOMER_EMAIL_UPDATED PosTransactionProperties.PosTransactionPropertyValue=true
I would like to create fields and values that look like the following for all events:
RECEIPT_EMAIL_ADDRESS = email@example.com RECEIPT_DELIVERY_METHOD = EMAIL+PAPER CUSTOMER_EMAIL_UPDATED = true
I am taking a three (3) step process to achieving my goal: (Still having issues on Step 3)
Step 1 - I did this by extracting the fields using some multivalue field extractions in props.conf and transforms.conf, putting them into two fields (postransactionproperties_PosTransactionPropertyCode and postransactionproperties_PosTransactionPropertyValue)
props.conf REPORT-trans_PosTransactionProperties = t_PTP_PosTransactionPropertyCode, t_PTP_PosTransactionPropertyValue transforms.conf # =-=- PosTransactionProperties.PosTransactionPropertyCode=RECEIPT_EMAIL_ADDRESS [t_PTP_PosTransactionPropertyCode] REGEX = PosTransactionProperties\[[0-9]+\]\.PosTransactionPropertyCode\[[0-9]+\]\=([^\n]+)\n FORMAT = postransactionproperties_PosTransactionPropertyCode::$1 MV_ADD = true REPEAT_MATCH = true # =-=- PosTransactionProperties.PosTransactionPropertyValuefirstname.lastname@example.org [t_PTP_PosTransactionPropertyValue] REGEX = PosTransactionProperties\[[0-9]+\]\.PosTransactionPropertyValue\[[0-9]+\]\=([^\n]+)\n FORMAT = postransactionproperties_PosTransactionPropertyValue::$1 MV_ADD = true REPEAT_MATCH = true
I then end up with the two mvfields, with all the values lines up in the order they show up in the event.
postransactionproperties_PosTransactionPropertyCode RECEIPT_EMAIL_ADDRESS RECEIPT_DELIVERY_METHOD CUSTOMER_EMAIL_UPDATED postransactionproperties_PosTransactionPropertyValue email@example.com EMAIL+PAPER true
Step 2 - I then combine the two fields in the order they exist, into a single field matching the values up together, with a separator using an eval statement in the props.conf file
props.conf EVAL-glue = mvzip(postransactionproperties_PosTransactionPropertyCode,postransactionproperties_PosTransactionPropertyValue,":::")
The result is a field named 'glue' that has values as such:
CUSTOMER_EMAIL_UPDATED:::true RECEIPT_DELIVERY_METHOD:::EMAIL+PAPER RECEIPT_EMAIL_ADDRESS:::firstname.lastname@example.org
Step 3 - Which I am having trouble with, I want to now break the values of the field 'glue' up into Key => Value pairs. I am thinking I can do this again with a props.conf and a transforms.conf statement as such:
props.conf REPORT-trans_glue = t_unglue transforms.conf [t_unglue] SOURCE_KEY = glue REGEX = ([a-zA-Z0-9\_\-]+):::([a-zA-Z0-9]+) FORMAT = $1::$2
This unfortunately does not work at all, and I need a little help to either understand why, and how to get it to work, or attack this whole problem in a different fashion all together. (NOTE: I understand the REGEX statement is not 100% correct, but it should have at least matched the 'true' result and it does not).
That seems a bit convoluted to me, is there a reason for not using key-value extraction capabilities in transforms.conf directly?
[your_sourcetype] TRANSFORMS-keyvalue = PosTransactionProperties ...
[PosTransactionProperties] REGEX = PosTransactionProperties\[\d+\]\.PosTransactionPropertyCode\[\d+\]=(?<_KEY_1>[^\n\r]+)[\n\r]+PosTransactionProperties\[\d+\]\.PosTransactionPropertyValue\[\d+\]=(?<_VAL_1>[^\n\r]+)
This will extract both the field name and the field value using the magic
_VAL_n names. Nothing to do in the search itself or in calculated fields, the fields will just be there automatically.
Edit: The reason for your approach not working out is that calculated fields (
EVAL-foo in props.conf) are applied after field extractions, so your transforms.conf rules don't see those fields.