How to write regex for below log to extract fields?

[vineela]

Hi All,

     I need to write regular expression for the below log to extract few fields. Can you please help me on that.
Here is the log:

{"log":"[14:38:36.117] [INFO ] [] [c.c.n.b.i.DefaultBusinessEventService] [akka://MmsAuCluster/system/sharding/notificationEnrichmentBpmn/0/oR6fulqKQOmr0axiUzCI2w_10/oR6fulqKQOmr0axiUzCI2w] - method=prepare; triggerName=creationCompleted, entity={'id'='2957b3205bf211ed8ded12d15e0c927a_1972381_29168b705bf211ed8ded12d15e0c927a','eventCode'='MDMT.MANDATE_CREATION_COMPLETED','paymentSystemId'='MMS','servicingAgentBIC'='null','messageIdentification'='2957b3205bf211ed8ded12d15e0c927a','businessDomainName'='Mandate','catalogCode'='MDMT','functionCode'='MANDATE_CREATION_COMPLETED','eventCodeDescription'='Mandate creation request completed','subjectEntityType'='MNDT','type'='MSG_DATA','dataFormat'='JSON','dataEncoding'='UTF-8','requestBody'='null''responseBody'='class ChannelNotification3 { mmsServicerBic: CTBAAUSNBKW trigger: MCRT priority: NORM mandateIdentification: 29168b705bf211ed8ded12d15e0c927a bulkIdentification: null reportIdentification: null actionIdentification: 2916b2805bf211ed8ded12d15e0c927a portingIdentification: null actionExpiryTime: null resolutionRequestedBy: null bulkItemResult: null }'} \n","stream":"stdout","docker":{"container_id":"1cbf6fee4ccb236146b7d66fd2f60e4d47c89012fba7679083141eb9a5342a94"},"kubernetes":{"container_name":"mms-au","namespace_name":"msaas-t4","pod_name":"mms-au-b-1-67d78896c6-c5t7s","container_image":"pso.docker.internal.cba/mms-au:2.3.2-0-1-ff0ef7b23","container_image_id":"docker-pullable://pso.docker.internal.cba/mms-au@sha256:cd39a1f76bb50314638a4b7642aa21d7280eca5923298db0b07df63a276bdd34","pod_id":"f649125d-2978-41ea-908f-f99aa84134f3","pod_ip":"100.64.85.236","host":"ip-10-3-197-109.ap-southeast-2.compute.internal","labels":{"app":"mms-au","dc":"b-1","pod-template-hash":"67d78896c6","release":"mms-au"},"master_url":"https://172.20.0.1:443/api","namespace_id":"48ee871a-7e60-45c4-b0f4-ee320a9512f5","namespace_labels":{"argocd.argoproj.io/instance":"appspaces","ci":"CM0953076","kubernetes.io/metadata.name":"msaas-t4","name":"msaas-t4","platform":"PSU","service_owner":"somersd","spg":"CBA_PAYMENTS_TEST_COORDINATION"}},"hostname":"ip-10-3-197-109.ap-southeast-2.compute.internal","host_ip":"10.3.197.109","cluster":"nonprod/pmn02"}

I need to extract fields called event code,trigger,mmsservicerbic - these 3 are highlighted above.as those are in different format and under log sub field i am not able to write. Can anyone help please

Thanks in Advance

[yuanliu]

While I usually advise not to use regex to extract from structured data, this particular log has many layers of escapement, so the data is JSON wrapped in key-value pair wrapped in JSON.  Still, for me retaining versatility and full dataset is worth trying.  Here goes. (Splunk should have already given you a field named "log".  The first spath is included in the following in case that is not the case.)

| spath ``` only if Splunk doesn't give you field "log" from the outer JSON ```
| rex field=log "entity={(?<entity>.*)}"
| rename entity as _raw ``` if you need the original _raw, first rename it temp ```
| rex mode=sed "s/'([^']+)'=/\1=/g"
| rex mode=sed "y/'/\"/" ``` reconstruct inner kv pairs ```
| kv
| fields eventCode responseBody ``` remove this line to retain all fields ```
| eval responseBody="{". replace(responseBody, ".*{ (.+) }", "\1") . "}"
| eval responseBody=replace(responseBody, "([^{\s]+):\s*([^\s}]+)", "\"\1\":\"\2\"")
| rex field=responseBody mode=sed "y/ /,/" ``` reconstruct inter JSON ```
| spath input=responseBody

The sample data would give something like

eventCode	actionExpiryTime	actionIdentification	bulkIdentification	bulkItemResult	mandateIdentification	mmsServiceBic	portingIdentification	priority	reportIdentification	resolutionRequestedBy	trigger
MDMT.MANDATE_CREATION_COMPLETED	null	2916b2805bf211ed8ded12d15e0c927a	null	null	29168b705bf211ed8ded12d15e0c927a	CTBAAUSNBKW	null	NORM	null	null	MCRT

[bowesmana]

Use this

| rex field=log "eventCode.=.(?<eventCode>[^']*)"
| rex field=log "mmsServicerBic:\s(?<mmsServicerBic>\w+)"
| rex field=log "trigger:\s(?<trigger>\w+)"

You may be able to shrink that into a single rex, but if the data changes, it may break