How to extract the fields in json format?

vineela — Tue, 21 Mar 2023 16:57:23 GMT

I am trying to extract the fields in json format. But not able to fetch the data.
PFB screenshot for reference:

not able to extract fields. Can anyone help on this.

Thanks in Advance.

Re: json format extraction

ITWhisperer — Tue, 21 Mar 2023 09:07:28 GMT

Please can you share your raw event in a code block </> rather than a picture, so that it can be used to test solutions?

Re: json format extraction

vineela — Tue, 21 Mar 2023 09:13:45 GMT

[1=javax.net.ssl.SSLHandshakeException[Remote host terminated the handshake],3=ip-10-31-39-168.ap-southeast-2.compute.internal/10.31.39.168:1417 (ip-10-31-39-168.ap-southeast-2.compute.internal),4=SSLSocket.startHandshake,5=default]\\n\tat com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.protocolConnect(RemoteTCPConnection.java:1325)\\n\tat com.ibm.mq.jmqi.remote.impl.RemoteConnection.connect(RemoteConnection.java:863)\\n\tat com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSessionFromNewConnection(RemoteConnectionSpecification.java:409)\\n\tat com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSession(RemoteConnectionSpecification.java:305)\\n\tat com.ibm.mq.jmqi.remote.impl.RemoteConnectionPool.getSession(RemoteConnectionPool.java:146)\\n\tat com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1730)\\n\t... 21 more\\nCaused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake\\n\tat sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1575)\\n\tat sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1405)\\n\tat sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1305)\\n\tat sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)\\n\tat com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection$6.run(RemoteTCPConnection.java:1294)\\n\tat com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection$6.run(RemoteTCPConnection.java:1286)\\n\tat java.security.AccessController.doPrivileged(Native Method)\\n\tat com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.protocolConnect(RemoteTCPConnection.java:1286)\\n\t... 26 more\\nCaused by: java.io.EOFException: SSL peer shut down incorrectly\\n\tat sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:167)\\n\tat sun.security.ssl.SSLTransport.decode(SSLTransport.java:109)\\n\tat sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1397)\\n\t... 32 more\\n java.io.EOFException: SSL peer shut down incorrectly\\n\tat sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:167)\\n\tat sun.security.ssl.SSLTransport.decode(SSLTransport.java:109)\\n\tat sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1397)\\n\t... 32 common frames omitted\\nWrapped by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake\\n\tat sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1575)\\n\tat sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1405)\\n\tat sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1305)\\n\tat sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)\\n\tat com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection$6.run(RemoteTCPConnection.java:1294)\\n\tat com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection$6.run(RemoteTCPConnection.java:1286)\\n\tat java.security.AccessController.doPrivileged(Native Method)\\n\tat com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.protocolConnect(RemoteTCPConnection.java:1286)\\n\t... 26 common frames omitted\\nWrapped by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2397;AMQ9771: SSL handshake failed. [1=javax.net.ssl.SSLHandshakeException[Remote host terminated the handshake],3=ip-10-31-39-168.ap-southeast-2.compute.internal/10.31.39.168:1417 (ip-10-31-39-168.ap-southeast-2.compute.internal),4=SSLSocket.startHandshake,5=default]\\n\tat com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.protocolConnect(RemoteTCPConnection.java:1325)\\n\tat com.ibm.mq.jmqi.remote.impl.RemoteConnection.connect(RemoteConnection.java:863)\\n\tat com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSessionFromNewConnection(RemoteConnectionSpecification.java:409)\\n\tat com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSession(RemoteConnectionSpecification.java:305)\\n\tat com.ibm.mq.jmqi.remote.impl.RemoteConnectionPool.getSession(RemoteConnectionPool.java:146)\\n\tat com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1730)\\n\t... 21 common frames omitted\\nWrapped by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2397;AMQ9204: Connection to host '10.31.39.168(1417)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2397;AMQ9771: SSL handshake failed. [1=javax.net.ssl.SSLHandshakeException[Remote host terminated the handshake],3=ip-10-31-39-168.ap-southeast-2.compute.internal/10.31.39.168:1417 (ip-10-31-39-168.ap-southeast-2.compute.internal),4=SSLSocket.startHandshake,5=default]],3=10.31.39.168(1417),5=RemoteTCPConnection.protocolConnect]\\n\tat com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:2282)\\n\tat com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1294)\\n\tat '10.31.39.168(1417)'.\\n\tat com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl._createConnection(JmsConnectionFactoryImpl.java:299)\\n\tat com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createConnection(JmsConnectionFactoryImpl.java:236)\\n\tat com.ibm.mq.jms.MQConnectionFactory.createCommonConnection(MQConnectionFactory.java:6024)\\n\tat com.ibm.mq.jms.MQConnectionFactory.createConnection(MQConnectionFactory.java:6049)\\n\tat org.springframework.jms.connection.SingleConnectionFactory.doCreateConnection(SingleConnectionFactory.java:410)\\n\tat org.springframework.jms.connection.SingleConnectionFactory.initConnection(SingleConnectionFactory.java:350)\\n\tat org.springframework.jms.connection.SingleConnectionFactory.getConnection(SingleConnectionFactory.java:328)\\n\tat org.springframework.jms.connection.SingleConnectionFactory.createConnection(SingleConnectionFactory.java:243)\\n\tat org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:196)\\n\tat org.springframework.jms.listener.AbstractJmsListeningContainer.createSharedConnection(AbstractJmsListeningContainer.java:412)\\n\tat {"container_id":"167559efc3f8cc23c1a2be1aa697978f1ec5092efc60b7f87b91f28a6f6dea76"},"kubernetes":{"container_name":"pj-npp-event-listener-psu-api","namespace_name":"msaas-badev","pod_name":"pj-npp-event-listener-psu-api-3.3.4.4-f57686594-q4shr","container_image":"pso.docker.internal.cba/pj-npp-jms-listener:3.3.4.4","container_image_id":"docker-pullable://pso.docker.internal.cba/pj-npp-jms-listener@sha256:83b20688216eb0ab4737123a03566c064bc565de4a89858978abdf4333b48ea0","pod_id":"8d7f77a2-0111-474d-82e2-c66103a4d807","pod_ip":"100.64.186.130","host":"ip-10-3-198-164.ap-southeast-2.compute.internal","labels":{"app":"pj-npp-event-listener","app.kubernetes.io/instance":"pj-npp-event-listener","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"pj-npp-event-listener","helm.sh/chart":"psu-api-1.7.0","heritage":"Helm","pod-template-hash":"f57686594","project":"pjpds","release":"pj-npp-event-listener"},"master_url":"https://172.20.0.1:443/api","namespace_id":"25c93690-5c3b-4f2b-a967-8d0355ea90f2","namespace_labels":{"argocd.argoproj.io/instance":"appspaces","ci":"CM0953076","kubernetes.io/metadata.name":"msaas-badev","name":"msaas-badev","platform":"PSU","service_owner":"somersd","spg":"CBA_PAYMENTS_TEST_COORDINATION"}},"hostname":"ip-10-3-198-164.ap-southeast-2.compute.internal","host_ip":"10.3.198.164","cluster":"nonprod/pmn02"}

Re: json format extraction

ITWhisperer — Tue, 21 Mar 2023 09:25:57 GMT

I guess I meant the full event

in a code block to preserve formatting

Having said that, you appear to only be trying to extract from the log field, has this been correctly extracted? Can you share an example? Is this extraction failing for all your events or only some of them?

Re: json format extraction

vineela — Tue, 21 Mar 2023 11:06:30 GMT

Yes, i need to extract errorcode and errormessage from log field which is in json format.

The log looks like this:
{"log":"21/Mar/2023:20:06:29 +1100 [defaultJmsListenerContainer-1] [correlationId=] ERROR au.com.commbank.pso.payments.pj.listener.util.LoggingUtil - Severity=ERROR, DateTimestamp=21/Mar/2023 20:06:29, ErrorCode=PJ_LISTENER_ERR_0003, ErrorMessage=PJ Listener connection to MQ has failed, MicroserviceName=PJ_LISTENER, ExceptionStackTrace=com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0018: Failed to connect to queue manager 'NPAT01' with connection mode 'Client' and host name '10.31.39.168(1417)'.\\nCheck the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information."}

I used this query to extract the data from this:
index="a0_payservutil_generic_app_audit_npd" sourcetype="cba:appinfra:hec:json" "PJ_LISTENER_ERR_000*"
| rename _raw as temp, log as _raw | kv | rename temp as _raw
|table ErrorCode ErrorMessage

Re: json format extraction

ITWhisperer — Tue, 21 Mar 2023 11:55:27 GMT

If this isn't working for you, it would seem to suggest that the log field has not been extracted.

In this example, representing your event, I have used spath to extract log from the _raw field before switching to with the _raw field to use kv

| makeresults | eval _raw="{\"log\":\"21/Mar/2023:20:06:29 +1100 [defaultJmsListenerContainer-1] [correlationId=] ERROR au.com.commbank.pso.payments.pj.listener.util.LoggingUtil - Severity=ERROR, DateTimestamp=21/Mar/2023 20:06:29, ErrorCode=PJ_LISTENER_ERR_0003, ErrorMessage=PJ Listener connection to MQ has failed, MicroserviceName=PJ_LISTENER, ExceptionStackTrace=com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0018: Failed to connect to queue manager 'NPAT01' with connection mode 'Client' and host name '10.31.39.168(1417)'.\\nCheck the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.\"}" | spath | rename _raw as temp, log as _raw | kv | rename temp as _raw | table ErrorCode ErrorMessage

topic Re: json format extraction in Splunk Search

How to extract the fields in json format?

Re: json format extraction

Re: json format extraction

Re: json format extraction

Re: json format extraction

Re: json format extraction