JSON Extraction of Values and Fields Logs Using SPATH or rex

[JossPRG]

Hello. I've been trying for days now and can't make the following work. Let me show you what I have.

My search looks like this:

index="yyy-privacy-apps" splunk_server_group=default2 sourcetype="yyy-priv-collector" source="/app/logs/yyy-privacy-apps/yyy-privacy-collector/yyy-priv-collector-perf-prod-us-west-2.log" b6815ff3-6742-4c4b-b69a-fbf99ddb24fb

 
What I want to do is to be able to extract several of the values from _raw into a table format. 

2025-08-27 16:14:15,006 ActivityGUID=931eecb0-8570-4045-8f83-d232065374ab TransactionGUID=9911aa1c-54bc-4912-8b4c-7b8d1b15434f ProductName=yyy-priv-collector HostLocal=10.177.774.774 ActivityName=endpoint OperationName=searchPayments RequestMethod=POST Duration=181 TimestampStart=2025-08-27 09:14:14,824 Timestamp=2025-08-27 09:14:15,005 DurationN=181 StatusCode=0 Client-ID=privCaseManagement HostRemote=127.0.0.6 TimestampEnd=2025-08-27 09:14:15,005 ResponseStatus=200 ActivityStep=rs 
@RequestURL=http://yyy-priv-collector-prod.rcp.deptandpriv.prod-cts.exp-aws.net/v1/searchPayments/
@ResponseHeaders={Content-Type=[application/json], X-Content-Type-Options=[nosniff], X-XSS-Protection=[0], Cache-Control=[no-cache, no-store, max-age=0, must-revalidate], Pragma=[no-cache], Expires=[0], X-Frame-Options=[DENY]}
@RequestHeaders={host=[yyy-priv-collector-prod.rcp.deptandpriv.prod-cts.exp-aws.net], accept=[application/json], transaction-guid=[9911aa1c-54bc-4912-8b4c-7b8d1b15434f], message-guid=[92d969c4-f659-45f4-b4ec-97a9f1e38e69], content-type=[application/json], client-id=[RCM], authorization=#SCRAMBLED#, x-datadog-trace-id=[1798983498446935042], x-datadog-parent-id=[1394352222710180060], x-datadog-sampling-priority=[0], content-length=[222], user-agent=[Apache-HttpClient/4.5.14 (Java/17.0.11)], accept-encoding=[gzip,deflate], x-forwarded-for=[10.172.234.74], x-forwarded-proto=[https], x-request-id=[9eb3657b-da6e-4f0e-84b9-41ca00dae160], x-envoy-attempt-count=[1], x-envoy-internal=[true], x-forwarded-client-cert=[By=spiffe://deptandpriv.prod-cts.exp-aws.net/ns/priv-tech-platform/sa/yyy-priv-collector-prod-us-west-2-prod-template;Hash=c2dc4556e3158a274d6eef6700c9fc5088993a732c77568766b809838dcb3e00;Subject="";URI=spiffe://deptandpriv.prod-cts.exp-aws.net/ns/istio-system/sa/istio-ingressgateway-service-account]}
@ResponsePayload={"errorInformation":null,"payments":[{"hits":[{"currentState":{"createdDate":"2025-08-26T16:33:51.899Z","entityDataVersion":3243315992,"entityId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|b6815ff3-6742-4c4b-b69a-fbf99ddb24fb|","entitySequenceId":"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c9pvrx-a6b87555-8ceb-480a-bea8-1697c5a6f9ca_132_4982","entityType":"state","lifecycleName":"CreditCardWith3DS","paymentTransactionId":"3e3cbd48-30ea-0565-2bb4-f7370681ecbf","previousTransactionStateId":null,"schemaVersion":"1.0","stateDate":"2025-08-26T16:33:51.899Z","stateName":"ChargeCompleted","stateTransition":null,"transactionStateId":"b6815ff3-6742-5c6b-b69a-fbf99ddb24fb","updatedDate":"2025-08-26T16:33:51.899Z"},"operations":{"abandonOperations":null,"captureDepositActivityOperations":null,"captureSettlementOperations":null,"chargeOperations":[{"attributes":{"additionalAttributes":null,"alternateCurrencyCode":null,"alternatePaymentAmount":null,"captureId":null,"createdDate":"2025-08-26T16:14:24.152Z","currencyCode":"USD","entityDataVersion":3243315992,"entityId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|e43db6f8-78a0-0e20-2636-d5b173280df3|","entitySequenceId":"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c9pvrx-a6b87555-8ceb-480a-bea8-1697c5a6f9ca_132_4982","entityType":"operation","operationTypeCode":"CHR","paymentAmount":944.6,"paymentOperationId":"e43db6f8-78a0-0e20-2636-d5b173280df3","paymentTransactionId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf","rateQuoteId":null,"rateQuoteServiceId":null,"refundId":null,"schemaVersion":"1.0","updatedDate":"2025-08-26T16:15:01.647Z"},"chargeSteps":null,"commitSteps":[{"additionalAttributes":null,"attemptId":0,"authorizationCode":null,"authorizationDate":"null","createdDate":"2025-08-26T16:15:01.582Z","currencyCode":"USD","entityDataVersion":3243315992,"entityId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|e43db6f8-78a0-0e20-2636-d5b173280df3|7d2f68fc-db1f-0238-2a35-8611bc8a90c4|","entitySequenceId":"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c9pvrx-a6b87555-8ceb-480a-bea8-1697c5a6f9ca_132_4982","entityType":"step","gatewayOperationId":"527ee823-4eba-4580-aa8c-cef59fc4a0db","gatewayResultStatuses":null,"gatewayResults":[],"gatewayService":"PaymentExecutor","internalReferenceId":null,"merchantId":null,"operationStepAmount":944.6,"operationStepPresentedAmount":944.6,"operationStepTransactedAmount":null,"operationStepTypeCode":"CMT","paymentOperationId":"e43db6f8-78a0-0e20-2636-d5b173280df3","paymentOperationStepId":"7d2f68fc-db1f-0238-2a35-8611bc8a90c4","paymentTransactionId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf","processor":null,"processorReferenceId":null,"schemaVersion":"1.0","statusCode":"0","statusCodeCategoryName":"Success","statusCodeMessageText":"Successfully processed!","statusCodeNamespace":"com.expedia.e3.es.payment.common.errorhandling.CommonStatusReportingTemplates","stepStatus":"success","successful":true,"updatedDate":"2025-08-26T16:15:01.637Z","partnerOperationId":null,"platformOperationId":null}],"holdSteps":[{"additionalAttributes":null,"attemptId":0,"authorizationCode":null,"authorizationDate":"null","createdDate":"2025-08-26T16:14:26.341Z","currencyCode":"USD","entityDataVersion":3243315992,"entityId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|e43db6f8-78a0-0e20-2636-d5b173280df3|793b4768-1ea0-07df-20d9-a280205711c1|","entitySequenceId":"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c9pvrx-a6b87555-8ceb-480a-bea8-1697c5a6f9ca_132_4982","entityType":"step","gatewayOperationId":"8859e2e7-6a85-4899-87ce-fb4bd7360944","gatewayResultStatuses":null,"gatewayResults":[],"gatewayService":"PaymentExecutor","internalReferenceId":null,"merchantId":null,"operationStepAmount":944.6,"operationStepPresentedAmount":944.6,"operationStepTransactedAmount":null,"operationStepTypeCode":"HLD","paymentOperationId":"e43db6f8-78a0-0e20-2636-d5b173280df3","paymentOperationStepId":"793b4768-1ea0-07df-20d9-a280205711c1","paymentTransactionId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf","processor":null,"processorReferenceId":null,"schemaVersion":"1.0","statusCode":"0","statusCodeCategoryName":"Success","statusCodeMessageText":"Successfully processed!","statusCodeNamespace":"com.expedia.e3.es.payment.common.errorhandling.CommonStatusReportingTemplates","stepStatus":"success","successful":true,"updatedDate":"2025-08-26T16:14:26.369Z","partnerOperationId":null,"platformOperationId":null}],"redirectSteps":null,"rollbackSteps":null}],"chargebackOperations":null,"chargebackReversalOperations":null,"chargebackReversalSettlementOperations":null,"chargebackSettlementOperations":null,"creditOperations":null,"refundDepositActivityOperations":null,"refundOperations":null,"refundSettlementOperations":null,"verifyOperations":[{"attributes":{"additionalAttributes":null,"alternateCurrencyCode":null,"alternatePaymentAmount":null,"captureId":null,"createdDate":"2025-08-26T16:14:24.116Z","currencyCode":"USD","entityDataVersion":3243315992,"entityId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|81025324-2e27-0aef-2fb3-ff5ab2a756c2|","entitySequenceId":"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c9pvrx-a6b87555-8ceb-480a-bea8-1697c5a6f9ca_132_4982","entityType":"operation","operationTypeCode":"VFY","paymentAmount":944.6,"paymentOperationId":"81025324-2e27-0aef-2fb3-ff5ab2a756c2","paymentTransactionId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf","rateQuoteId":null,"rateQuoteServiceId":null,"refundId":null,"schemaVersion":"1.0","updatedDate":"2025-08-26T16:14:25.331Z"},"verifySteps":[{"additionalAttributes":null,"attemptId":0,"authorizationCode":"ul425i9o0ur","authorizationDate":"2025-08-26T16:14:25.031Z","createdDate":"2025-08-26T16:14:24.199Z","currencyCode":"USD","entityDataVersion":3243270965,"entityId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|81025324-2e27-0aef-2fb3-ff5ab2a756c2|e2e1bd80-a27e-031e-25b7-87c5369ae5e1|","entitySequenceId":"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c6p5jw-ed84cd28-fe3f-47e0-a445-139c6a6b8dca_132_8899","entityType":"step","gatewayOperationId":"cf737c29-d68d-4c99-9d39-1be40b5e84fe","gatewayResultStatuses":[{"acquirerStatusCode":null,"description":null,"errorResultType":null,"internalReferenceId":"o98kxkapu3y","merchantId":"EXPEDIAAUTH","paymentProcessorStatusCodeType":null,"processor":"Bibit","processorReferenceId":"ul425i9o0ur","ptxGatewayResultStatuses":[{"acquirerStatusCode":null,"description":"The acquirer responds that the address details partially match the details at the issuer.","errorResultType":"Technical","ptxGatewayResultStatusTypeCode":"avs","rawStatusCode":"PARTIAL_APPROVED","statusCategory":"EXP","statusCode":"3"},{"acquirerStatusCode":null,"description":"The acquirer responds that the CVC code matches the details at the issuer","errorResultType":"Technical","ptxGatewayResultStatusTypeCode":"cvv","rawStatusCode":"APPROVED","statusCategory":"EXP","statusCode":"7"},{"acquirerStatusCode":null,"description":"Authorized","errorResultType":"None","ptxGatewayResultStatusTypeCode":"gateway","rawStatusCode":"AUTHORISED","statusCategory":"EXTERNAL","statusCode":"SUCCESSFUL"}],"rawStatusCode":null,"statusCategory":null,"statusCode":null}],"gatewayResults":[],"gatewayService":"PaymentExecutor","internalReferenceId":"o98kxkapu3y","merchantId":"EXPEDIAAUTH","operationStepAmount":944.6,"operationStepPresentedAmount":944.6,"operationStepTransactedAmount":0.0,"operationStepTypeCode":"VFY","paymentOperationId":"81025324-2e27-0aef-2fb3-ff5ab2a756c2","paymentOperationStepId":"e2e1bd80-a27e-031e-25b7-87c5369ae5e1","paymentTransactionId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf","processor":"Bibit","processorReferenceId":"ul425i9o0ur","reusedVerificationId":"487f7891-06f9-4e01-bb91-a34cbdecc6fb","reusing":false,"schemaVersion":"1.0","statusCode":"0","statusCodeCategoryName":"Success","statusCodeMessageText":"MSTERR_NO_ERROR","statusCodeNamespace":"com.expedia.e3.es.payment.processor.HAPSResponseCode.MSTE","stepStatus":"success","successful":true,"updatedDate":"2025-08-26T16:14:25.277Z"}]}]
},"stateHistory":[{"createdDate":"2025-08-26T16:16:01.534Z","entityDataVersion":3243274925,"entityId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|fa9cfff3-0cd5-49e7-a583-ca9965d75f82|","entitySequenceId":"pay-payin-executor-service-prod-us-west-2-prod-primary-2-cplwgn-98f56f82-560c-4b69-b18e-86e1c1ff7aeb_151_9275","entityType":"state","lifecycleName":"CreditCardWith3DS","paymentTransactionId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf","previousTransactionStateId":null,"schemaVersion":"1.0","stateDate":"2025-08-26T16:16:01.534Z","stateName":"ChargeCompleted","stateTransition":null,"transactionStateId":"fa9cfff3-0cd5-49e7-a583-ca9965d75f82","updatedDate":"2025-08-26T16:16:01.534Z"},{"createdDate":"2025-08-26T16:14:25.385Z","entityDataVersion":3243270965,"entityId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|12713ce5-6e3a-4d79-8804-64de8383abaa|","entitySequenceId":"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c6p5jw-ed84cd28-fe3f-47e0-a445-139c6a6b8dca_132_8899","entityType":"state","lifecycleName":"CreditCardWith3DS","paymentTransactionId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf","previousTransactionStateId":"45a30bc0-6aae-496d-a45f-f03afd458eaf","schemaVersion":"1.0","stateDate":"2025-08-26T16:14:25.385Z","stateName":"Verified","stateTransition":"New:Verified","transactionStateId":"12713ce5-6e3a-4d79-8804-64de8383abaa","updatedDate":"2025-08-26T16:14:25.385Z"},{"createdDate":"2025-08-26T16:14:26.479Z","entityDataVersion":3243271007,"entityId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|6aee2039-cf71-440a-9c60-d49c7d34342f|","entitySequenceId":"pay-payin-executor-service-prod-us-west-2-prod-primary-2-cvnv9c-e3efcc74-a5e0-437f-84d3-f5e1691dbf67_157_4357","entityType":"state","lifecycleName":"CreditCardWith3DS","paymentTransactionId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf","previousTransactionStateId":"d1f42d9c-9e3b-46e4-82a4-c14bef915f8e","schemaVersion":"1.0","stateDate":"2025-08-26T16:14:26.479Z","stateName":"HoldCompleted","stateTransition":"Verified:HoldCompleted","transactionStateId":"6aee2039-cf71-440a-9c60-d49c7d34342f","updatedDate":"2025-08-26T16:14:26.479Z"},{"createdDate":"2025-08-26T16:15:01.752Z","entityDataVersion":3243272446,"entityId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|3deafd0f-f794-4535-9fa5-ae6c76efe0da|","entitySequenceId":"pay-payin-executor-service-prod-us-west-2-prod-primary-2-cplwgn-98f56f82-560c-4b69-b18e-86e1c1ff7aeb_151_9068","entityType":"state","lifecycleName":"CreditCardWith3DS","paymentTransactionId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf","previousTransactionStateId":"833dec67-6be9-4e2b-a76c-00b677963ee6","schemaVersion":"1.0","stateDate":"2025-08-26T16:15:01.752Z","stateName":"ChargePending","stateTransition":"HoldCompleted:ChargePending","transactionStateId":"3deafd0f-f794-4535-9fa5-ae6c76efe0da","updatedDate":"2025-08-26T16:15:01.752Z"},{"createdDate":"2025-08-26T16:14:25.385Z","entityDataVersion":3243270965,"entityId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|45a30bc0-6aae-496d-a45f-f03afd458eaf|","entitySequenceId":"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c6p5jw-ed84cd28-fe3f-47e0-a445-139c6a6b8dca_132_8899","entityType":"state","lifecycleName":"CreditCardWith3DS","paymentTransactionId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf","previousTransactionStateId":null,"schemaVersion":"1.0","stateDate":"2025-08-26T16:14:25.385Z","stateName":"New","stateTransition":null,"transactionStateId":"45a30bc0-6aae-496d-a45f-f03afd458eaf","updatedDate":"2025-08-26T16:14:25.385Z"},{"createdDate":"2025-08-26T16:15:01.752Z","entityDataVersion":3243272446,"entityId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|75ca49ea-eb88-4473-b029-75a55846048d|","entitySequenceId":"pay-payin-executor-service-prod-us-west-2-prod-primary-2-cplwgn-98f56f82-560c-4b69-b18e-86e1c1ff7aeb_151_9068","entityType":"state","lifecycleName":"CreditCardWith3DS","paymentTransactionId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf","previousTransactionStateId":"3deafd0f-f794-4535-9fa5-ae6c76efe0da","schemaVersion":"1.0","stateDate":"2025-08-26T16:15:01.752Z","stateName":"ChargeCompleted","stateTransition":"ChargePending:ChargeCompleted","transactionStateId":"75ca49ea-eb88-4473-b029-75a55846048d","updatedDate":"2025-08-26T16:15:01.752Z"},{"createdDate":"2025-08-26T16:33:51.899Z","entityDataVersion":3243315992,"entityId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|b6815ff3-6742-4c4b-b69a-fbf99ddb24fb|","entitySequenceId":"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c9pvrx-a6b87555-8ceb-480a-bea8-1697c5a6f9ca_132_4982","entityType":"state","lifecycleName":"CreditCardWith3DS","paymentTransactionId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf","previousTransactionStateId":null,"schemaVersion":"1.0","stateDate":"2025-08-26T16:33:51.899Z","stateName":"ChargeCompleted","stateTransition":null,"transactionStateId":"b6815ff3-6742-4c4b-b69a-fbf99ddb24fb","updatedDate":"2025-08-26T16:33:51.899Z"}],"transaction":{"bankId":null,"bankIdentificationNumber":"403476","billingMerchantCode":"ThirdParty","bookingDirectoryParentId":null,"brandName":"Visa","businessLiabilityCategory":null,"chargeId":null,"cardPresentCode":"WebPaymentByCustomer","clientRequestGuid":"80224b87-a1ce-4fcf-8e03-8f6fad71d73c:100000","clientTransactionId":"e35c6097-5829-4c0c-9c3a-57f7c03b34d7","collectingLegalEntity":null,"companyCode":"10126","createdDate":"2025-08-26T16:14:24.103Z","eapid":"0","entityDataVersion":3243315992,"entityId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|","entitySequenceId":"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c9pvrx-a6b87555-8ceb-480a-bea8-1697c5a6f9ca_132_4982","entityType":"transaction","expediaPurchaseTypeId":"21","gpid":"0","instrumentVerificationDataId":"6d7532f0-5a2b-4105-bf2c-a4d3a8f3ff8c","instrumentVerificationType":"cvv","itineraryBookingDirectoryId":null,"jurisdiction":"USA","languageCode":"en","lastFourDigits":"7288","lifeCycleModelVersion":"V1","locale":"en_US","managementUnit":"1255","mandateAcceptanceDate":"null","mandateId":null,"mandateType":null,"numberOfInstallments":null,"orderId":"-9223371999231674674","orderNumber":"9076832532564","orderOperationCorrelationId":null,"paymentAllocationRecordRefCode":null,"paymentDescription":"Itinerary: 73221117050482, Start Date: 10/7/2025, End Date: 10/8/2025, Contains: Agency Air","paymentEndDate":"null","paymentInstrumentId":"90a9fca1-b36e-7710-f8f2-fc03bb9871f3","paymentInstrumentService":"PV","paymentIntentId":null,"paymentItemCorrelationId":"e1849640-1f57-4ba8-8ac3-25f73d63b118","paymentPlanCorrelationId":"f6957b2b-8a54-4725-8cea-16a0fd58985a","paymentPlanId":"a3e6a9e0-43a7-x67b-af2a-4ec0aeeb06ab","paymentProcessorClientGuid":"payorch","paymentProviderId":"12","paymentReasonCode":null,"paymentReasonId":"0","paymentSubType":"Visa","paymentTransactionId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf","paymentType":"CreditCard","pointOfSaleOrderReferenceNumber":"73221117050482","proxyForPayerId":"0","scheduledDate":"null","schemaVersion":"1.0","siteId":null,"transactionModelName":"TwoStepCommit","transferType":null,"travServerDbInstance":"TravServerUS","travelProductId":"80001","trl":"421159069","tuid":"744159015","tuidLogon":"744159015","updatedDate":"2025-08-26T16:15:01.658Z","partnerTransactionId":null,"platformTransactionId":null,"partnerAccountId":null,"paymentPlanRoutingId":"ern:pay:ptx:r2::60566b68-85c4-0a43-27df-169fc5cb285f"},"transactionAttributes":{"descriptorAttributes":[{"attributeName":"DESCRIPTOR","attributeTypeCode":"DESCRIPTOR","createdDate":"null","descriptorPhone":"tvly.com ","descriptorText":"TRAVELOCITY*73221117050482","entityDataVersion":3243270965,"entityId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|DESCRIPTOR|","entitySequenceId":"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c6p5jw-ed84cd28-fe3f-47e0-a445-139c6a6b8dca_132_8899","entityType":"attribute","paymentTransactionId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf","schemaVersion":"1.0","updatedDate":"2025-08-26T16:14:25.385Z"}],"stringAttributes":[{"attributeName":"PaymentProviderID","attributeTypeCode":"STRING","createdDate":"null","entityDataVersion":3243315992,"entityId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|PaymentProviderID|","entitySequenceId":"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c9pvrx-a6b87555-8ceb-480a-bea8-1697c5a6f9ca_132_4982","entityType":"attribute","paymentTransactionId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf","schemaVersion":"1.0","updatedDate":"2025-08-26T16:33:51.899Z","value":"12"},{"attributeName":"SupplierMerchantName","attributeTypeCode":"STRING","createdDate":"null","entityDataVersion":3243315992,"entityId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|SupplierMerchantName|","entitySequenceId":"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c9pvrx-a6b87555-8ceb-480a-bea8-1697c5a6f9ca_132_4982","entityType":"attribute","paymentTransactionId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf","schemaVersion":"1.0","updatedDate":"2025-08-26T16:33:51.899Z","value":"British Airways"},{"attributeName":"LegacyPaymentType","attributeTypeCode":"STRING","createdDate":"null","entityDataVersion":3243315992,"entityId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|LegacyPaymentType|","entitySequenceId":"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c9pvrx-a6b87555-8ceb-480a-bea8-1697c5a6f9ca_132_4982","entityType":"attribute","paymentTransactionId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf","schemaVersion":"1.0","updatedDate":"2025-08-26T16:33:51.899Z","value":"Full"}],"taggingAttributes":[{"attributeName":"ManagementUnit","attributeTypeCode":"TAGGING","createdDate":"null","entityDataVersion":3243315992,"entityId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|ManagementUnit|","entitySequenceId":"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c9pvrx-a6b87555-8ceb-480a-bea8-1697c5a6f9ca_132_4982","entityType":"attribute","paymentTransactionId":"5e5cbc48-29ea-0565-2bb4-f7370681ecbf","schemaVersion":"1.0","updatedDate":"2025-08-26T16:33:51.899Z","value":"1255"}],"threeDSAttributes":null}}]
,"scrollId":null,"totalHits":1}],"paymentInstruments":[{"paymentInstrumentID":"90A9FCA1-B36E-7710-F8F2-FC03BB9871F3","paymentMethod":"BankIssuedCard","paymentSubMethod":"Visa","brandName":"Visa","cardNumber":null,"cardType":"CREDIT","token":"XXXX","instrumentVerificationType":null,"instrumentDescription":null,"expirationDate":"XXXX","bin":"403476","last4Digits":"XXXX","bankRoutingNumber":null,"bankAccountNumber":null,"bankId":null,"bankBranchCode":null,"mandateType":null,"mandateID":null,"mandateAcceptanceDate":null,"pin":null,"accountId":null,"linkedPaymentInstruments":null,"customerInfo":{"personName":{"personalTitle":null,"firstName":"XXXX","middleName":null,"lastName":"XXXX","suffixName":null},"phoneNumber":{"phoneCategoryCode":null,"phoneCountryCode":"XXXX","phoneAreaCode":"XXXX","phoneNumber":"XXXX","phoneExtensionNumber":null},"address":{"addressCategoryCode":null,"companyNameAddressLine":null,"firstAddressLine":"XXXX","secondAddressLine":null,"thirdAddressLine":null,"fourthAddressLine":null,"fifthAddressLine":null,"cityName":"XXXX","provinceName":"XXXX","postalCode":"XXXX","countryCode":"XXXX","personName":null,"phoneNumber":null,"addressStatus":null},"emailAddress":"XXXX","taxId":null,"taxIdType":null,"payerId":null,"payerStatus":null,"payerCountry":null,"payerBusiness":null,"cardHolderName":null},"links":null,"presentedFormOfPayment":null}]}
@RequestPayload={"paymentInstrumentId":null,"paymentIntentIds":null,"itineraryNumber":"73221117050482","travelProductId":"80001","merchantOrderCode":null,"tuid":null,"internalReferenceId":null,"acquirerReferenceNumber":null,"userId":null}

From the section that begins with "@ResponsePayload" I would like to extract the values for
"createdDate"
"paymentTransactionId"
"currencyCode"
"operationStepAmount"

And towards the end of _raw I would like to extract the values for
"itineraryNumber"
"travelProductId"

I have tried using spath (for example the following and many other variations):
| spath input=ResponsePayload
| rename ResponsePayload{}.ftid as message
| table message _raw

I have tried using rex (for example the following and many other variations):
| rex "(?P<json_field>(.*ResponsePayload={.*}))"

I have read several threads and resources on spath and rex since those seem to be the most promising, but I've got none to work. I keep getting _raw or empty fields or no results.

Thanks in advance for your advice

[yuanliu]

Unfortunately, ResponsePayload field would not be normally extracted.  There are multiple challenges with this one.  One is sheer size of the raw events.  This may be overcome by tweaking limits.conf if you have control of it.  Then, many field values are not properly quoted for automatic extraction.  Using custom regex extraction in props.conf may help.  But the irregular use of line breaks (at least as shown in the illustration) can make this very difficult.

In addition, you need to have some prior knowledge about the actual dataset in relation to the desired results. Keys "createdDate", "paymentTransactionId", "currencyCode", "operationStepAmount" are as you noted, inside the JSON field ResonsePayload.  But they are part of an array of arrays.  How do you want to handle array elements when there are multiple?  ResponsePayload has another array, paymentInstruments.  How do you want to handle multiple elements in this field?  Keys "itineraryNumber" and "travelProductId" are in a different JSON field RequestPayload, which is simpler to extract.  However, this field is also not automatically extracted due to lack of proper quotes.

In the following, I will assume that elements of ResponsePayload.paymentInstruments should always be retained as an array, but each element of ResponsePayload.payments, and each element of ResponsePayload.payments.hits should be viewed individually.  The method used to break array elements, mvexpand, has its limits and can have performance consequences if the arrays are too big.

| rex "@RequestPayload=(?<RequestPayload>.+)"
| spath input=RequestPayload
``` RequestPayload is very easy to extract ```
| rex mode=sed "s/.+ActivityGUID=.+\n//
 s/@.+(URL|Headers)=.+\n//g
 s/@RequestPayload=.+//
 s/@ResponsePayload=//"
``` the complex sed is needed because of irregular line breaks within RequestPayload ```
| rename _raw as ResponsePayload
| spath input=ResponsePayload errorInformation
| spath input=ResponsePayload paymentInstruments{}
| spath input=ResponsePayload payments{}
| fields - ResponsePayload
| mvexpand payments{}
| spath input=payments{} scrollId
| spath input=payments{} totalHits
| spath input=payments{} hits{}
| fields - payments{}
| mvexpand hits{}
| spath input=hits{}
| fields - hits{}

In the above, I have already examined keys in those arrays, and assume that the keys are always the same.  If there can be more than 3 keys in each array, or if keys are indeterministic, you can use a more traditional, conservative approach

| rex "@RequestPayload=(?<RequestPayload>.+)"
| spath input=RequestPayload
``` RequestPayload is very easy to extract ```
| rex mode=sed "s/.+ActivityGUID=.+\n//
 s/@.+(URL|Headers)=.+\n//g
 s/@RequestPayload=.+//
 s/@ResponsePayload=//"
``` the complex sed is needed because of irregular line breaks within RequestPayload ```
| rename _raw as ResponsePayload
| spath input=ResponsePayload
| fields - payments{}.*
| spath input=ResponsePayload payments{}
| fields - ResponsePayload
| mvexpand payments{}
| spath input=payments{}
| fields - hits{}.*
| spath input=payments{} hits{}
| fields - payments{}
| mvexpand hits{}
| spath input=hits{}
| fields - hits{}

Here is the emulation I use.  Play with it and compare with real data

| makeresults
| fields - _time
| eval _raw = "2025-08-27 16:14:15,006 ActivityGUID=931eecb0-8570-4045-8f83-d232065374ab TransactionGUID=9911aa1c-54bc-4912-8b4c-7b8d1b15434f ProductName=yyy-priv-collector HostLocal=10.177.774.774 ActivityName=endpoint OperationName=searchPayments RequestMethod=POST Duration=181 TimestampStart=2025-08-27 09:14:14,824 Timestamp=2025-08-27 09:14:15,005 DurationN=181 StatusCode=0 Client-ID=privCaseManagement HostRemote=127.0.0.6 TimestampEnd=2025-08-27 09:14:15,005 ResponseStatus=200 ActivityStep=rs 
@RequestURL=http://yyy-priv-collector-prod.rcp.deptandpriv.prod-cts.exp-aws.net/v1/searchPayments/
@ResponseHeaders={Content-Type=[application/json], X-Content-Type-Options=[nosniff], X-XSS-Protection=[0], Cache-Control=[no-cache, no-store, max-age=0, must-revalidate], Pragma=[no-cache], Expires=[0], X-Frame-Options=[DENY]}
@RequestHeaders={host=[yyy-priv-collector-prod.rcp.deptandpriv.prod-cts.exp-aws.net], accept=[application/json], transaction-guid=[9911aa1c-54bc-4912-8b4c-7b8d1b15434f], message-guid=[92d969c4-f659-45f4-b4ec-97a9f1e38e69], content-type=[application/json], client-id=[RCM], authorization=#SCRAMBLED#, x-datadog-trace-id=[1798983498446935042], x-datadog-parent-id=[1394352222710180060], x-datadog-sampling-priority=[0], content-length=[222], user-agent=[Apache-HttpClient/4.5.14 (Java/17.0.11)], accept-encoding=[gzip,deflate], x-forwarded-for=[10.172.234.74], x-forwarded-proto=[https], x-request-id=[9eb3657b-da6e-4f0e-84b9-41ca00dae160], x-envoy-attempt-count=[1], x-envoy-internal=[true], x-forwarded-client-cert=[By=spiffe://deptandpriv.prod-cts.exp-aws.net/ns/priv-tech-platform/sa/yyy-priv-collector-prod-us-west-2-prod-template;Hash=c2dc4556e3158a274d6eef6700c9fc5088993a732c77568766b809838dcb3e00;Subject=\"\";URI=spiffe://deptandpriv.prod-cts.exp-aws.net/ns/istio-system/sa/istio-ingressgateway-service-account]}
@ResponsePayload={\"errorInformation\":null,\"payments\":[{\"hits\":[{\"currentState\":{\"createdDate\":\"2025-08-26T16:33:51.899Z\",\"entityDataVersion\":3243315992,\"entityId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|b6815ff3-6742-4c4b-b69a-fbf99ddb24fb|\",\"entitySequenceId\":\"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c9pvrx-a6b87555-8ceb-480a-bea8-1697c5a6f9ca_132_4982\",\"entityType\":\"state\",\"lifecycleName\":\"CreditCardWith3DS\",\"paymentTransactionId\":\"3e3cbd48-30ea-0565-2bb4-f7370681ecbf\",\"previousTransactionStateId\":null,\"schemaVersion\":\"1.0\",\"stateDate\":\"2025-08-26T16:33:51.899Z\",\"stateName\":\"ChargeCompleted\",\"stateTransition\":null,\"transactionStateId\":\"b6815ff3-6742-5c6b-b69a-fbf99ddb24fb\",\"updatedDate\":\"2025-08-26T16:33:51.899Z\"},\"operations\":{\"abandonOperations\":null,\"captureDepositActivityOperations\":null,\"captureSettlementOperations\":null,\"chargeOperations\":[{\"attributes\":{\"additionalAttributes\":null,\"alternateCurrencyCode\":null,\"alternatePaymentAmount\":null,\"captureId\":null,\"createdDate\":\"2025-08-26T16:14:24.152Z\",\"currencyCode\":\"USD\",\"entityDataVersion\":3243315992,\"entityId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|e43db6f8-78a0-0e20-2636-d5b173280df3|\",\"entitySequenceId\":\"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c9pvrx-a6b87555-8ceb-480a-bea8-1697c5a6f9ca_132_4982\",\"entityType\":\"operation\",\"operationTypeCode\":\"CHR\",\"paymentAmount\":944.6,\"paymentOperationId\":\"e43db6f8-78a0-0e20-2636-d5b173280df3\",\"paymentTransactionId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf\",\"rateQuoteId\":null,\"rateQuoteServiceId\":null,\"refundId\":null,\"schemaVersion\":\"1.0\",\"updatedDate\":\"2025-08-26T16:15:01.647Z\"},\"chargeSteps\":null,\"commitSteps\":[{\"additionalAttributes\":null,\"attemptId\":0,\"authorizationCode\":null,\"authorizationDate\":\"null\",\"createdDate\":\"2025-08-26T16:15:01.582Z\",\"currencyCode\":\"USD\",\"entityDataVersion\":3243315992,\"entityId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|e43db6f8-78a0-0e20-2636-d5b173280df3|7d2f68fc-db1f-0238-2a35-8611bc8a90c4|\",\"entitySequenceId\":\"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c9pvrx-a6b87555-8ceb-480a-bea8-1697c5a6f9ca_132_4982\",\"entityType\":\"step\",\"gatewayOperationId\":\"527ee823-4eba-4580-aa8c-cef59fc4a0db\",\"gatewayResultStatuses\":null,\"gatewayResults\":[],\"gatewayService\":\"PaymentExecutor\",\"internalReferenceId\":null,\"merchantId\":null,\"operationStepAmount\":944.6,\"operationStepPresentedAmount\":944.6,\"operationStepTransactedAmount\":null,\"operationStepTypeCode\":\"CMT\",\"paymentOperationId\":\"e43db6f8-78a0-0e20-2636-d5b173280df3\",\"paymentOperationStepId\":\"7d2f68fc-db1f-0238-2a35-8611bc8a90c4\",\"paymentTransactionId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf\",\"processor\":null,\"processorReferenceId\":null,\"schemaVersion\":\"1.0\",\"statusCode\":\"0\",\"statusCodeCategoryName\":\"Success\",\"statusCodeMessageText\":\"Successfully processed!\",\"statusCodeNamespace\":\"com.expedia.e3.es.payment.common.errorhandling.CommonStatusReportingTemplates\",\"stepStatus\":\"success\",\"successful\":true,\"updatedDate\":\"2025-08-26T16:15:01.637Z\",\"partnerOperationId\":null,\"platformOperationId\":null}],\"holdSteps\":[{\"additionalAttributes\":null,\"attemptId\":0,\"authorizationCode\":null,\"authorizationDate\":\"null\",\"createdDate\":\"2025-08-26T16:14:26.341Z\",\"currencyCode\":\"USD\",\"entityDataVersion\":3243315992,\"entityId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|e43db6f8-78a0-0e20-2636-d5b173280df3|793b4768-1ea0-07df-20d9-a280205711c1|\",\"entitySequenceId\":\"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c9pvrx-a6b87555-8ceb-480a-bea8-1697c5a6f9ca_132_4982\",\"entityType\":\"step\",\"gatewayOperationId\":\"8859e2e7-6a85-4899-87ce-fb4bd7360944\",\"gatewayResultStatuses\":null,\"gatewayResults\":[],\"gatewayService\":\"PaymentExecutor\",\"internalReferenceId\":null,\"merchantId\":null,\"operationStepAmount\":944.6,\"operationStepPresentedAmount\":944.6,\"operationStepTransactedAmount\":null,\"operationStepTypeCode\":\"HLD\",\"paymentOperationId\":\"e43db6f8-78a0-0e20-2636-d5b173280df3\",\"paymentOperationStepId\":\"793b4768-1ea0-07df-20d9-a280205711c1\",\"paymentTransactionId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf\",\"processor\":null,\"processorReferenceId\":null,\"schemaVersion\":\"1.0\",\"statusCode\":\"0\",\"statusCodeCategoryName\":\"Success\",\"statusCodeMessageText\":\"Successfully processed!\",\"statusCodeNamespace\":\"com.expedia.e3.es.payment.common.errorhandling.CommonStatusReportingTemplates\",\"stepStatus\":\"success\",\"successful\":true,\"updatedDate\":\"2025-08-26T16:14:26.369Z\",\"partnerOperationId\":null,\"platformOperationId\":null}],\"redirectSteps\":null,\"rollbackSteps\":null}],\"chargebackOperations\":null,\"chargebackReversalOperations\":null,\"chargebackReversalSettlementOperations\":null,\"chargebackSettlementOperations\":null,\"creditOperations\":null,\"refundDepositActivityOperations\":null,\"refundOperations\":null,\"refundSettlementOperations\":null,\"verifyOperations\":[{\"attributes\":{\"additionalAttributes\":null,\"alternateCurrencyCode\":null,\"alternatePaymentAmount\":null,\"captureId\":null,\"createdDate\":\"2025-08-26T16:14:24.116Z\",\"currencyCode\":\"USD\",\"entityDataVersion\":3243315992,\"entityId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|81025324-2e27-0aef-2fb3-ff5ab2a756c2|\",\"entitySequenceId\":\"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c9pvrx-a6b87555-8ceb-480a-bea8-1697c5a6f9ca_132_4982\",\"entityType\":\"operation\",\"operationTypeCode\":\"VFY\",\"paymentAmount\":944.6,\"paymentOperationId\":\"81025324-2e27-0aef-2fb3-ff5ab2a756c2\",\"paymentTransactionId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf\",\"rateQuoteId\":null,\"rateQuoteServiceId\":null,\"refundId\":null,\"schemaVersion\":\"1.0\",\"updatedDate\":\"2025-08-26T16:14:25.331Z\"},\"verifySteps\":[{\"additionalAttributes\":null,\"attemptId\":0,\"authorizationCode\":\"ul425i9o0ur\",\"authorizationDate\":\"2025-08-26T16:14:25.031Z\",\"createdDate\":\"2025-08-26T16:14:24.199Z\",\"currencyCode\":\"USD\",\"entityDataVersion\":3243270965,\"entityId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|81025324-2e27-0aef-2fb3-ff5ab2a756c2|e2e1bd80-a27e-031e-25b7-87c5369ae5e1|\",\"entitySequenceId\":\"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c6p5jw-ed84cd28-fe3f-47e0-a445-139c6a6b8dca_132_8899\",\"entityType\":\"step\",\"gatewayOperationId\":\"cf737c29-d68d-4c99-9d39-1be40b5e84fe\",\"gatewayResultStatuses\":[{\"acquirerStatusCode\":null,\"description\":null,\"errorResultType\":null,\"internalReferenceId\":\"o98kxkapu3y\",\"merchantId\":\"EXPEDIAAUTH\",\"paymentProcessorStatusCodeType\":null,\"processor\":\"Bibit\",\"processorReferenceId\":\"ul425i9o0ur\",\"ptxGatewayResultStatuses\":[{\"acquirerStatusCode\":null,\"description\":\"The acquirer responds that the address details partially match the details at the issuer.\",\"errorResultType\":\"Technical\",\"ptxGatewayResultStatusTypeCode\":\"avs\",\"rawStatusCode\":\"PARTIAL_APPROVED\",\"statusCategory\":\"EXP\",\"statusCode\":\"3\"},{\"acquirerStatusCode\":null,\"description\":\"The acquirer responds that the CVC code matches the details at the issuer\",\"errorResultType\":\"Technical\",\"ptxGatewayResultStatusTypeCode\":\"cvv\",\"rawStatusCode\":\"APPROVED\",\"statusCategory\":\"EXP\",\"statusCode\":\"7\"},{\"acquirerStatusCode\":null,\"description\":\"Authorized\",\"errorResultType\":\"None\",\"ptxGatewayResultStatusTypeCode\":\"gateway\",\"rawStatusCode\":\"AUTHORISED\",\"statusCategory\":\"EXTERNAL\",\"statusCode\":\"SUCCESSFUL\"}],\"rawStatusCode\":null,\"statusCategory\":null,\"statusCode\":null}],\"gatewayResults\":[],\"gatewayService\":\"PaymentExecutor\",\"internalReferenceId\":\"o98kxkapu3y\",\"merchantId\":\"EXPEDIAAUTH\",\"operationStepAmount\":944.6,\"operationStepPresentedAmount\":944.6,\"operationStepTransactedAmount\":0.0,\"operationStepTypeCode\":\"VFY\",\"paymentOperationId\":\"81025324-2e27-0aef-2fb3-ff5ab2a756c2\",\"paymentOperationStepId\":\"e2e1bd80-a27e-031e-25b7-87c5369ae5e1\",\"paymentTransactionId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf\",\"processor\":\"Bibit\",\"processorReferenceId\":\"ul425i9o0ur\",\"reusedVerificationId\":\"487f7891-06f9-4e01-bb91-a34cbdecc6fb\",\"reusing\":false,\"schemaVersion\":\"1.0\",\"statusCode\":\"0\",\"statusCodeCategoryName\":\"Success\",\"statusCodeMessageText\":\"MSTERR_NO_ERROR\",\"statusCodeNamespace\":\"com.expedia.e3.es.payment.processor.HAPSResponseCode.MSTE\",\"stepStatus\":\"success\",\"successful\":true,\"updatedDate\":\"2025-08-26T16:14:25.277Z\"}]}]
},\"stateHistory\":[{\"createdDate\":\"2025-08-26T16:16:01.534Z\",\"entityDataVersion\":3243274925,\"entityId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|fa9cfff3-0cd5-49e7-a583-ca9965d75f82|\",\"entitySequenceId\":\"pay-payin-executor-service-prod-us-west-2-prod-primary-2-cplwgn-98f56f82-560c-4b69-b18e-86e1c1ff7aeb_151_9275\",\"entityType\":\"state\",\"lifecycleName\":\"CreditCardWith3DS\",\"paymentTransactionId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf\",\"previousTransactionStateId\":null,\"schemaVersion\":\"1.0\",\"stateDate\":\"2025-08-26T16:16:01.534Z\",\"stateName\":\"ChargeCompleted\",\"stateTransition\":null,\"transactionStateId\":\"fa9cfff3-0cd5-49e7-a583-ca9965d75f82\",\"updatedDate\":\"2025-08-26T16:16:01.534Z\"},{\"createdDate\":\"2025-08-26T16:14:25.385Z\",\"entityDataVersion\":3243270965,\"entityId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|12713ce5-6e3a-4d79-8804-64de8383abaa|\",\"entitySequenceId\":\"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c6p5jw-ed84cd28-fe3f-47e0-a445-139c6a6b8dca_132_8899\",\"entityType\":\"state\",\"lifecycleName\":\"CreditCardWith3DS\",\"paymentTransactionId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf\",\"previousTransactionStateId\":\"45a30bc0-6aae-496d-a45f-f03afd458eaf\",\"schemaVersion\":\"1.0\",\"stateDate\":\"2025-08-26T16:14:25.385Z\",\"stateName\":\"Verified\",\"stateTransition\":\"New:Verified\",\"transactionStateId\":\"12713ce5-6e3a-4d79-8804-64de8383abaa\",\"updatedDate\":\"2025-08-26T16:14:25.385Z\"},{\"createdDate\":\"2025-08-26T16:14:26.479Z\",\"entityDataVersion\":3243271007,\"entityId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|6aee2039-cf71-440a-9c60-d49c7d34342f|\",\"entitySequenceId\":\"pay-payin-executor-service-prod-us-west-2-prod-primary-2-cvnv9c-e3efcc74-a5e0-437f-84d3-f5e1691dbf67_157_4357\",\"entityType\":\"state\",\"lifecycleName\":\"CreditCardWith3DS\",\"paymentTransactionId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf\",\"previousTransactionStateId\":\"d1f42d9c-9e3b-46e4-82a4-c14bef915f8e\",\"schemaVersion\":\"1.0\",\"stateDate\":\"2025-08-26T16:14:26.479Z\",\"stateName\":\"HoldCompleted\",\"stateTransition\":\"Verified:HoldCompleted\",\"transactionStateId\":\"6aee2039-cf71-440a-9c60-d49c7d34342f\",\"updatedDate\":\"2025-08-26T16:14:26.479Z\"},{\"createdDate\":\"2025-08-26T16:15:01.752Z\",\"entityDataVersion\":3243272446,\"entityId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|3deafd0f-f794-4535-9fa5-ae6c76efe0da|\",\"entitySequenceId\":\"pay-payin-executor-service-prod-us-west-2-prod-primary-2-cplwgn-98f56f82-560c-4b69-b18e-86e1c1ff7aeb_151_9068\",\"entityType\":\"state\",\"lifecycleName\":\"CreditCardWith3DS\",\"paymentTransactionId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf\",\"previousTransactionStateId\":\"833dec67-6be9-4e2b-a76c-00b677963ee6\",\"schemaVersion\":\"1.0\",\"stateDate\":\"2025-08-26T16:15:01.752Z\",\"stateName\":\"ChargePending\",\"stateTransition\":\"HoldCompleted:ChargePending\",\"transactionStateId\":\"3deafd0f-f794-4535-9fa5-ae6c76efe0da\",\"updatedDate\":\"2025-08-26T16:15:01.752Z\"},{\"createdDate\":\"2025-08-26T16:14:25.385Z\",\"entityDataVersion\":3243270965,\"entityId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|45a30bc0-6aae-496d-a45f-f03afd458eaf|\",\"entitySequenceId\":\"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c6p5jw-ed84cd28-fe3f-47e0-a445-139c6a6b8dca_132_8899\",\"entityType\":\"state\",\"lifecycleName\":\"CreditCardWith3DS\",\"paymentTransactionId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf\",\"previousTransactionStateId\":null,\"schemaVersion\":\"1.0\",\"stateDate\":\"2025-08-26T16:14:25.385Z\",\"stateName\":\"New\",\"stateTransition\":null,\"transactionStateId\":\"45a30bc0-6aae-496d-a45f-f03afd458eaf\",\"updatedDate\":\"2025-08-26T16:14:25.385Z\"},{\"createdDate\":\"2025-08-26T16:15:01.752Z\",\"entityDataVersion\":3243272446,\"entityId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|75ca49ea-eb88-4473-b029-75a55846048d|\",\"entitySequenceId\":\"pay-payin-executor-service-prod-us-west-2-prod-primary-2-cplwgn-98f56f82-560c-4b69-b18e-86e1c1ff7aeb_151_9068\",\"entityType\":\"state\",\"lifecycleName\":\"CreditCardWith3DS\",\"paymentTransactionId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf\",\"previousTransactionStateId\":\"3deafd0f-f794-4535-9fa5-ae6c76efe0da\",\"schemaVersion\":\"1.0\",\"stateDate\":\"2025-08-26T16:15:01.752Z\",\"stateName\":\"ChargeCompleted\",\"stateTransition\":\"ChargePending:ChargeCompleted\",\"transactionStateId\":\"75ca49ea-eb88-4473-b029-75a55846048d\",\"updatedDate\":\"2025-08-26T16:15:01.752Z\"},{\"createdDate\":\"2025-08-26T16:33:51.899Z\",\"entityDataVersion\":3243315992,\"entityId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|b6815ff3-6742-4c4b-b69a-fbf99ddb24fb|\",\"entitySequenceId\":\"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c9pvrx-a6b87555-8ceb-480a-bea8-1697c5a6f9ca_132_4982\",\"entityType\":\"state\",\"lifecycleName\":\"CreditCardWith3DS\",\"paymentTransactionId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf\",\"previousTransactionStateId\":null,\"schemaVersion\":\"1.0\",\"stateDate\":\"2025-08-26T16:33:51.899Z\",\"stateName\":\"ChargeCompleted\",\"stateTransition\":null,\"transactionStateId\":\"b6815ff3-6742-4c4b-b69a-fbf99ddb24fb\",\"updatedDate\":\"2025-08-26T16:33:51.899Z\"}],\"transaction\":{\"bankId\":null,\"bankIdentificationNumber\":\"403476\",\"billingMerchantCode\":\"ThirdParty\",\"bookingDirectoryParentId\":null,\"brandName\":\"Visa\",\"businessLiabilityCategory\":null,\"chargeId\":null,\"cardPresentCode\":\"WebPaymentByCustomer\",\"clientRequestGuid\":\"80224b87-a1ce-4fcf-8e03-8f6fad71d73c:100000\",\"clientTransactionId\":\"e35c6097-5829-4c0c-9c3a-57f7c03b34d7\",\"collectingLegalEntity\":null,\"companyCode\":\"10126\",\"createdDate\":\"2025-08-26T16:14:24.103Z\",\"eapid\":\"0\",\"entityDataVersion\":3243315992,\"entityId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|\",\"entitySequenceId\":\"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c9pvrx-a6b87555-8ceb-480a-bea8-1697c5a6f9ca_132_4982\",\"entityType\":\"transaction\",\"expediaPurchaseTypeId\":\"21\",\"gpid\":\"0\",\"instrumentVerificationDataId\":\"6d7532f0-5a2b-4105-bf2c-a4d3a8f3ff8c\",\"instrumentVerificationType\":\"cvv\",\"itineraryBookingDirectoryId\":null,\"jurisdiction\":\"USA\",\"languageCode\":\"en\",\"lastFourDigits\":\"7288\",\"lifeCycleModelVersion\":\"V1\",\"locale\":\"en_US\",\"managementUnit\":\"1255\",\"mandateAcceptanceDate\":\"null\",\"mandateId\":null,\"mandateType\":null,\"numberOfInstallments\":null,\"orderId\":\"-9223371999231674674\",\"orderNumber\":\"9076832532564\",\"orderOperationCorrelationId\":null,\"paymentAllocationRecordRefCode\":null,\"paymentDescription\":\"Itinerary: 73221117050482, Start Date: 10/7/2025, End Date: 10/8/2025, Contains: Agency Air\",\"paymentEndDate\":\"null\",\"paymentInstrumentId\":\"90a9fca1-b36e-7710-f8f2-fc03bb9871f3\",\"paymentInstrumentService\":\"PV\",\"paymentIntentId\":null,\"paymentItemCorrelationId\":\"e1849640-1f57-4ba8-8ac3-25f73d63b118\",\"paymentPlanCorrelationId\":\"f6957b2b-8a54-4725-8cea-16a0fd58985a\",\"paymentPlanId\":\"a3e6a9e0-43a7-x67b-af2a-4ec0aeeb06ab\",\"paymentProcessorClientGuid\":\"payorch\",\"paymentProviderId\":\"12\",\"paymentReasonCode\":null,\"paymentReasonId\":\"0\",\"paymentSubType\":\"Visa\",\"paymentTransactionId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf\",\"paymentType\":\"CreditCard\",\"pointOfSaleOrderReferenceNumber\":\"73221117050482\",\"proxyForPayerId\":\"0\",\"scheduledDate\":\"null\",\"schemaVersion\":\"1.0\",\"siteId\":null,\"transactionModelName\":\"TwoStepCommit\",\"transferType\":null,\"travServerDbInstance\":\"TravServerUS\",\"travelProductId\":\"80001\",\"trl\":\"421159069\",\"tuid\":\"744159015\",\"tuidLogon\":\"744159015\",\"updatedDate\":\"2025-08-26T16:15:01.658Z\",\"partnerTransactionId\":null,\"platformTransactionId\":null,\"partnerAccountId\":null,\"paymentPlanRoutingId\":\"ern:pay:ptx:r2::60566b68-85c4-0a43-27df-169fc5cb285f\"},\"transactionAttributes\":{\"descriptorAttributes\":[{\"attributeName\":\"DESCRIPTOR\",\"attributeTypeCode\":\"DESCRIPTOR\",\"createdDate\":\"null\",\"descriptorPhone\":\"tvly.com \",\"descriptorText\":\"TRAVELOCITY*73221117050482\",\"entityDataVersion\":3243270965,\"entityId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|DESCRIPTOR|\",\"entitySequenceId\":\"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c6p5jw-ed84cd28-fe3f-47e0-a445-139c6a6b8dca_132_8899\",\"entityType\":\"attribute\",\"paymentTransactionId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf\",\"schemaVersion\":\"1.0\",\"updatedDate\":\"2025-08-26T16:14:25.385Z\"}],\"stringAttributes\":[{\"attributeName\":\"PaymentProviderID\",\"attributeTypeCode\":\"STRING\",\"createdDate\":\"null\",\"entityDataVersion\":3243315992,\"entityId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|PaymentProviderID|\",\"entitySequenceId\":\"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c9pvrx-a6b87555-8ceb-480a-bea8-1697c5a6f9ca_132_4982\",\"entityType\":\"attribute\",\"paymentTransactionId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf\",\"schemaVersion\":\"1.0\",\"updatedDate\":\"2025-08-26T16:33:51.899Z\",\"value\":\"12\"},{\"attributeName\":\"SupplierMerchantName\",\"attributeTypeCode\":\"STRING\",\"createdDate\":\"null\",\"entityDataVersion\":3243315992,\"entityId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|SupplierMerchantName|\",\"entitySequenceId\":\"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c9pvrx-a6b87555-8ceb-480a-bea8-1697c5a6f9ca_132_4982\",\"entityType\":\"attribute\",\"paymentTransactionId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf\",\"schemaVersion\":\"1.0\",\"updatedDate\":\"2025-08-26T16:33:51.899Z\",\"value\":\"British Airways\"},{\"attributeName\":\"LegacyPaymentType\",\"attributeTypeCode\":\"STRING\",\"createdDate\":\"null\",\"entityDataVersion\":3243315992,\"entityId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|LegacyPaymentType|\",\"entitySequenceId\":\"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c9pvrx-a6b87555-8ceb-480a-bea8-1697c5a6f9ca_132_4982\",\"entityType\":\"attribute\",\"paymentTransactionId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf\",\"schemaVersion\":\"1.0\",\"updatedDate\":\"2025-08-26T16:33:51.899Z\",\"value\":\"Full\"}],\"taggingAttributes\":[{\"attributeName\":\"ManagementUnit\",\"attributeTypeCode\":\"TAGGING\",\"createdDate\":\"null\",\"entityDataVersion\":3243315992,\"entityId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf|ManagementUnit|\",\"entitySequenceId\":\"pay-payin-executor-service-prod-us-west-2-prod-primary-2-c9pvrx-a6b87555-8ceb-480a-bea8-1697c5a6f9ca_132_4982\",\"entityType\":\"attribute\",\"paymentTransactionId\":\"5e5cbc48-29ea-0565-2bb4-f7370681ecbf\",\"schemaVersion\":\"1.0\",\"updatedDate\":\"2025-08-26T16:33:51.899Z\",\"value\":\"1255\"}],\"threeDSAttributes\":null}}]
,\"scrollId\":null,\"totalHits\":1}],\"paymentInstruments\":[{\"paymentInstrumentID\":\"90A9FCA1-B36E-7710-F8F2-FC03BB9871F3\",\"paymentMethod\":\"BankIssuedCard\",\"paymentSubMethod\":\"Visa\",\"brandName\":\"Visa\",\"cardNumber\":null,\"cardType\":\"CREDIT\",\"token\":\"XXXX\",\"instrumentVerificationType\":null,\"instrumentDescription\":null,\"expirationDate\":\"XXXX\",\"bin\":\"403476\",\"last4Digits\":\"XXXX\",\"bankRoutingNumber\":null,\"bankAccountNumber\":null,\"bankId\":null,\"bankBranchCode\":null,\"mandateType\":null,\"mandateID\":null,\"mandateAcceptanceDate\":null,\"pin\":null,\"accountId\":null,\"linkedPaymentInstruments\":null,\"customerInfo\":{\"personName\":{\"personalTitle\":null,\"firstName\":\"XXXX\",\"middleName\":null,\"lastName\":\"XXXX\",\"suffixName\":null},\"phoneNumber\":{\"phoneCategoryCode\":null,\"phoneCountryCode\":\"XXXX\",\"phoneAreaCode\":\"XXXX\",\"phoneNumber\":\"XXXX\",\"phoneExtensionNumber\":null},\"address\":{\"addressCategoryCode\":null,\"companyNameAddressLine\":null,\"firstAddressLine\":\"XXXX\",\"secondAddressLine\":null,\"thirdAddressLine\":null,\"fourthAddressLine\":null,\"fifthAddressLine\":null,\"cityName\":\"XXXX\",\"provinceName\":\"XXXX\",\"postalCode\":\"XXXX\",\"countryCode\":\"XXXX\",\"personName\":null,\"phoneNumber\":null,\"addressStatus\":null},\"emailAddress\":\"XXXX\",\"taxId\":null,\"taxIdType\":null,\"payerId\":null,\"payerStatus\":null,\"payerCountry\":null,\"payerBusiness\":null,\"cardHolderName\":null},\"links\":null,\"presentedFormOfPayment\":null}]}
@RequestPayload={\"paymentInstrumentId\":null,\"paymentIntentIds\":null,\"itineraryNumber\":\"73221117050482\",\"travelProductId\":\"80001\",\"merchantOrderCode\":null,\"tuid\":null,\"internalReferenceId\":null,\"acquirerReferenceNumber\":null,\"userId\":null}"

[PickleRick]

What you got there is a perfect example of what the data should _not_ look like.

You have an ugly mix of structured data and a "kinda-key-value" pairs additionally packed with an extra timestamp header.

You can't reasonably manipulate json data with regexes. And you can't reliably extract the json part from that message. You can try but there will be edge cases where it will fail.

Your best bet would be to get in touch with the team responsible for exporting the data and ask them to write the whole event as well-formed json structure.

[ITWhisperer · Accepted solution]

Your rex doesn't look quite right. Try something like this

| rex "ResponsePayload=(?<json_field>.*})"
| spath input=json_field

[JossPRG]

Thank you, changing the rex as you suggested made the difference.

I have been able to extract fields/values of interest by carefully following the brackets, commas, and quotations

It is looking something like this now and so far seems that I'm being able to extract what's needed. 
I do want to build some search commands to look for specific conditions, but that's a separate step:

index="yyy-privacy-apps" splunk_server_group=default3 sourcetype="yyy-search-service" source="/app/logs/yyy-privacy-apps/efr-search-service/yyy-search-service-perf.log" "73226186156856"

| rex "ResponsePayload=(?<json_field>.*})"
| fields - _raw
| spath input=json_field
| dedup json_field
| table _time
resultSet{}.itineraryNumber
resultSet{}.totalPrice.usdValue
json_field

[isoutamo]

On left side “interesting fields” with verbose search mode, did you have field name ResponsePayload or @ResponsePayload on it? If there are then you could try to use something like 

|spath input=ResponsePayload

to get it as a json. But this needs that content of this field is real json format without errors! Basically you could take content of this field and test it with jq or some other tool to ensure it.

If you haven’t this field extracted then again check that this is real json and also that it’s content doesn’t have any additional characters in source which breaks json format. I have seen e.g. that “ mark has replaced \u0022 which means that those are strings not jsons.