Wanted to ask one last time, did the above example help to show the data we are collecting and if it will be split right with the auto kv_mode? ... View more

Did the above example help to show the issue we are having? Hoping to get more clarification on this issue. ... View more

Here is the data example: Oct 10 22:53:38 server.com com.MetricsForwarder message_type=raw, TEAM=INGEST, version=2.0, environment=xxx, message="{\"header\": {\"namespace\":\"xxxx\",\"route-definition-id\":\"mobile-device-id\",\"job-instance-id\":\"123456789\",\"step-name\":\"\",\"step-instance-id\":\"123456789\",\"job-status\":\"SUCCESS\",\"step-status\":\"SUCCESS\",\"message-timestamp\":123456789} ,\"payload\":\"{ n \\\"namespace\\\" : \\\"xx-xxxxx\\\", n \\\"name\\\" : \\\"xxxx\\\", n \\\"internal-name\\\" : \\\"xxxx\\\", n \\\"version\\\" : \\\"1.0\\\", n \\\"state\\\" : \\\"published\\\", n \\\"category-internal-name\\\" : \\\"id-mapping\\\", n \\\"provider-id\\\" : \\\"1234\\\", n \\\"provider-name\\\" : \\\"XXXX\\\", n \\\"input\\\" : {\\n \\\"category-internal-name\\\" : \\\"input\\\",\\n \\\"input-uris\\\" : [ \\\"xx:\/\/xxxxxxxxxx-xxxxx\/prod\/data\/data\/input\/4312\/20161005\/mobile.123456789.DL.send\\\" ]\\n } , n \\\"output\\\" : [ {\\n \\\"category-internal-name\\\" : \\\"id-mapping\\\",\\n \\\"output-uris\\\" : [ \\\"nnn:\/\/xxxxx-xxxxx\/prod\/etl\/xxx\/xxxx\/device_processing\/data_canonical\/4312\/xx_xxxxx\/123456789\/123456789\\\" ]\\n } ], n \\\"update-timestamp\\\" : 123456789, n \\\"status\\\" : \\\"success\\\", n \\\"instance-id\\\" : \\\"application_123456789\\\", n \\\"input-validation\\\" : { n \\\"threshold\\\" : { n \\\"exceeds-threshold\\\" : false, n \\\"exceeds-drop-threshold\\\" : false, n \\\"exceeds-warn-threshold\\\" : false, n \\\"percent\\\" : {\\n \\\"drop-percent\\\" : 100,\\n \\\"warn-percent\\\" : 100\\n } n }, n \\\"summary\\\" : {\\n \\\"total-processed\\\" : 95,\\n \\\"total-pass\\\" : 95,\\n \\\"total-drop\\\" : 0,\\n \\\"total-warn\\\" : 0,\\n \\\"total-filter\\\" : 0\\n } , n \\\"failed-validations\\\" : [ ] n }, n \\\"output-validation\\\" : { n \\\"threshold\\\" : { n \\\"exceeds-threshold\\\" : false, n \\\"exceeds-drop-threshold\\\" : false, n \\\"exceeds-warn-threshold\\\" : false, n \\\"percent\\\" : {\\n \\\"drop-percent\\\" : 5,\\n \\\"warn-percent\\\" : 5\\n } n }, n \\\"summary\\\" : {\\n \\\"total-processed\\\" : 95,\\n \\\"total-pass\\\" : 95,\\n \\\"total-drop\\\" : 0,\\n \\\"total-warn\\\" : 0,\\n \\\"total-filter\\\" : 0\\n } , n \\\"failed-validations\\\" : [ ] n }, n \\\"output-profile\\\" : { n \\\"id\\\" : 1, n \\\"internalName\\\" : \\\"xxxxxxxxxx\\\", n \\\"name\\\" : \\\"xxxx XXXX xxxx xxxxx\\\", n \\\"profileDefintionUri\\\" : null, n \\\"dataSetUri\\\" : null, n \\\"schema\\\" : null, n \\\"variables\\\" : null, n \\\"metrics\\\" : [ { n \\\"id\\\" : 2, n \\\"displaySequence\\\" : 2, n \\\"internalName\\\" : \\\"xxxx_x_x_xxxx_xx_xxxx\\\", n \\\"name\\\" : \\\"Generic comment\\\", n \\\"metrics\\\" : [ { n \\\"dimensions\\\" : {\\n \\\"device_id_type\\\" : \\\"1\\\"\\n } , n \\\"value\\\" : 92 n } ] n }, { n \\\"id\\\" : 1, n \\\"displaySequence\\\" : 1, n \\\"internalName\\\" : \\\"xxxx_x_xx_xxxx\\\", n \\\"name\\\" : \\\"Generic field\\\", n \\\"metrics\\\" : [ {\\n \\\"dimensions\\\" : null,\\n \\\"value\\\" : 92\\n } ] n }, { n \\\"id\\\" : 1, n \\\"displaySequence\\\" : 0, n \\\"internalName\\\" : \\\"xxxx_x_xx_xxxx\\\", n \\\"name\\\" : \\\"Generic Field 2\\\", n \\\"metrics\\\" : [ {\\n \\\"dimensions\\\" : null,\\n \\\"value\\\" : 95\\n } ] n } ] n }, n \\\"process-counts\\\" : {\\n \\\"input\\\" : 95,\\n \\\"input-validation-input\\\" : 95,\\n \\\"input-validation-output\\\" : 95,\\n \\\"transform-input\\\" : 95,\\n \\\"transform-output\\\" : 95,\\n \\\"output-validation-input\\\" : 95,\\n \\\"output-validation-output\\\" : 95,\\n \\\"output-id-mapping\\\" : 95\\n } \\n}\"}" ... View more