parse the message json within log json

[venkateshn2382]

I have a log like below displayed in SPlunk UI. I want the "message" key to be parsed into json as well. how to do that?

The below is the raw text.

{"stream":"stderr","logtag":"F","message":"{\"Context\":{\"SourceTransactionID\":\"UMV-626036c8-b843-46e8-8ef3-0bd78376bf93\",\"CaseID\":\"UMV-UMV_OK_CAAS_MMR_Mokcup_PIPE_2023-11-28-151036894\",\"CommunicationID\":\"UMV-64b9c2a9-be74-4ec6-9fd0-f545c1dd890f\",\"RequestID\":\"4ea2b9be-752b-4e6f-8972-0c435d1ad282\",\"RecordID\":\"332ebe12-0269-4ae6-90fc-98c8887e3703\"},\"LogCollection\":[{\"source\":\"handler.go:44\",\"timestamp\":\"2023-11-30T15:01:07.209285695Z\",\"msg\":{\"specversion\":\"1.0\",\"type\":\"com.cnc.caas.documentgenerationservices.documentgeneration.completed.public\",\"source\":\"/events/caas/documentgenerationservices/record/documentgeneration\",\"id\":\"Rec#332ebe12-0269-4ae6-90fc-98c8887e3703\",\"time\":\"2023-11-30T15:01:06.972071059Z\",\"subject\":\"record-documentgenerationservices-wip\",\"dataschema\":\"/caas/comp_01_a_events-spec.json\",\"datacontenttype\":\"application/json\",\"data\":{\"CAAS\":{\"Event\":{\"Version\":\"2.0.0\",\"EventType\":\"documentgeneration.completed\",\"LifeCycleStatus\":\"wip\",\"EventSequence\":4,\"OriginTimeStamp\":\"2023-11-30T15:01:06.972Z\",\"SourceName\":\"UMV\",\"SourceTransactionID\":\"UMV-626036c8-b843-46e8-8ef3-0bd78376bf93\",\"CaseID\":\"UMV-UMV_OK_CAAS_MMR_Mokcup_PIPE_2023-11-28-151036894\",\"CommunicationID\":\"UMV-64b9c2a9-be74-4ec6-9fd0-f545c1dd890f\",\"RequestID\":\"4ea2b9be-752b-4e6f-8972-0c435d1ad282\",\"RecordID\":\"332ebe12-0269-4ae6-90fc-98c8887e3703\",\"RequestedDeliveryChannel\":\"Print\",\"RecordedDeliveryChannel\":\"Print\",\"AdditionalData\":{\"CompositionAttributes\":{\"IsOCOENotificationRequired\":true,\"JobID\":47130}},\"S3Location\":{\"BucketName\":\"cnc-caas-csl-dev-smartcomm-output\",\"ObjectKey\":\"output/4ea2b9be-752b-4e6f-8972-0c435d1ad282/47130/4ea2b9be-752b-4e6f-8972-0c435d1ad282_332ebe12-0269-4ae6-90fc-98c8887e3703_UMV-64b9c2a9-be74-4ec6-9fd0-f545c1dd890f_Payload.json\"},\"Priority\":false,\"EventFailedStatus\":0,\"RetryCount\":1,\"Errors\":null,\"OriginalSqsMessage\":{\"data\":{\"CAAS\":{\"Event\":{\"AdditionalData\":{\"CompositionAttributes\":{\"IsOCOENotificationRequired\":true,\"JobID\":47130}},\"CaseID\":\"UMV-UMV_OK_CAAS_MMR_Mokcup_PIPE_2023-11-28-151036894\",\"CommunicationGroupID\":\"mbrmatreqok\",\"CommunicationID\":\"UMV-64b9c2a9-be74-4ec6-9fd0-f545c1dd890f\",\"Errors\":null,\"EventFailedStatus\":0,\"EventSequence\":4,\"EventType\":\"recordcomposition.response.start\",\"LifeCycleStatus\":\"wip\",\"OriginTimeStamp\":\"2023-11-30T15:00:04.996Z\",\"PreRendered\":false,\"Priority\":false,\"RecipientID\":\"68032561\",\"RecipientType\":\"Member\",\"RecordID\":\"332ebe12-0269-4ae6-90fc-98c8887e3703\",\"RecordedDeliveryChannel\":\"Print\",\"RequestID\":\"4ea2b9be-752b-4e6f-8972-0c435d1ad282\",\"RequestedDeliveryChannel\":\"Print\",\"RetryCount\":1,\"S3Location\":{\"BucketName\":\"cnc-caas-csl-dev-smartcomm-output\",\"ObjectKey\":\"output/4ea2b9be-752b-4e6f-8972-0c435d1ad282/47130/4ea2b9be-752b-4e6f-8972-0c435d1ad282_332ebe12-0269-4ae6-90fc-98c8887e3703_UMV-64b9c2a9-be74-4ec6-9fd0-f545c1dd890f_Payload.json\"},\"SourceName\":\"UMV\",\"SourceTransactionID\":\"UMV-626036c8-b843-46e8-8ef3-0bd78376bf93\",\"Version\":\"2.0.0\"}}},\"datacontenttype\":\"application/json\",\"dataschema\":\"/caas/comp_01_a_events-spec.json\",\"id\":\"Rec#332ebe12-0269-4ae6-90fc-98c8887e3703\",\"source\":\"/events/caas/smart/record/composition\",\"specversion\":\"1.0\",\"subject\":\"record-composition-response-start\",\"time\":\"2023-11-30T15:01:05.756937686Z\",\"type\":\"com.cnc.caas.composition.response.start.private\"},\"CommunicationGroupID\":\"mbrmatreqok\",\"RecipientID\":\"68032561\",\"RecipientType\":\"Member\",\"PreRendered\":false}}}}},{\"source\":\"handler.go:46\",\"timestamp\":\"2023-11-30T15:01:07.21572506Z\",\"msg\":\"mongo insert id is 6568a3b3ab042d54478ef071\"}],\"RetryCount\":1,\"level\":\"error\",\"msg\":\"Log collector output\",\"time\":\"2023-11-30T15:01:07Z\"}","kubernetes":{"pod_name":"eventsupdatetomongo-d98bb8594-cnbsd","namespace_name":"caas-composition-layer","pod_id":"50d49842-793a-41c8-a903-11c23607dfd6","labels":{"app":"eventsupdatetomongo","pod-template-hash":"d98bb8594","version":"dcode-801-1.0.1-2745653"},"annotations":{"cattle.io/timestamp":"2023-06-08T22:30:33Z","cni.projectcalico.org/containerID":"58cf3b42ab43fac0a5bf1f97e5a4a7db9dbf6a572705f02480384e63c2a53288","cni.projectcalico.org/podIP":"172.17.224.31/32","cni.projectcalico.org/podIPs":"172.17.224.31/32","kubectl.kubernetes.io/restartedAt":"2023-11-20T17:28:31Z"},"host":"ip-10-168-125-122.ec2.internal","container_name":"eventsupdatetomongo","docker_id":"c83dd87422fbdcae60a40ac50bcad0f387d50f3021975b81dbccac1bc0d965b2","container_hash":"artifactory-aws.centene.com/caas-docker_non-production_local_aws/eventsupdatetomongo@sha256:3b7e5e0908cec3f68baa7f9be18397b6ce4aa807f92b98b6b8970edac9780388","container_image":"artifactory-aws.centene.com/caas-docker_non-production_local_aws/eventsupdatetomongo:dcode-801-1.0.1-2745653"}}

[venkateshn2382]

@gcusello The data is ingested via http event collector. 

[gcusello]

Hi @venkateshn2382 ,

you have to try to use the INDEXED_EXTRACTIONS = json in the sourcetype associated to the HEC.

Ciao.

Giuseppe 

[venkateshn2382]

@gcusello  will this work with Splunk cloud as well? is this option available? 

[gcusello]

Hi @venkateshn2382 ,

Yes, it will work.

Ciao.

Giuseppe

[gcusello]

hi @venkateshn2382 ,

did you tried to use INDEXED_EXTRACTIONS = json in your props.conf?

you can find more details at https://docs.splunk.com/Documentation/Splunk/Latest/Admin/Propsconf 

This option must be located in the Universal Forwarder and in the Heavy Forwarder (if present) and in the Search Heads.

Ciao.

Giuseppe