Hi All,
I have got 2 set of logs, one of which has the Connector details and the other has got the error details if any connector gets failed.
First set:
Log1: sink.http.apac.hk.dna.eap.a...
See more...
Hi All,
I have got 2 set of logs, one of which has the Connector details and the other has got the error details if any connector gets failed.
First set:
Log1: sink.http.apac.hk.dna.eap.adobedmp.derived.int.aamrtevents FAILED|mwgcb-csrla02u RUNNING|mwgcb-csrla02u NA
Log2: sink.http.apac.au.dna.eap.adobedmp.derived.int.aamrtevents RUNNING|mwgcb-csrla02u RUNNING|mwgcb-csrla02u NA
Log3: sink.http.apac.ph.dna.eap.adobedmp.derived.int.aamrtevents FAILED|mwgcb-csrla01u FAILED|mwgcb-csrla01u NA
Log4: sink.http.apac.th.dna.eap.adobedmp.derived.int.aamrtevents FAILED|swgcb-csrla01u FAILED|mwgcb-csrla01u NA
Here I used the below query to extract the required fields:
... | rex field=_raw "(\w+\.)(?P<Component>\w+)\." | rex field=_raw "(?P<Connector_Name>(\w+\.){3,12}\w+)\s" | rex field=_raw "(?P<Connector_Name>(\w+\-){3,12}\w+)\s" | rex field=_raw "(\w+\.){3,12}\w+\s(?P<Connector_State>\w+)\|" | rex field=_raw "(\w+\-){3,12}\w+\s(?P<Connector_State>\w+)\|" | rex field=_raw "(\w+\.){3,12}\w+\s\w+\|(?P<Worker_ID>\w+\-\w+)\s" | rex field=_raw "(\w+\-){3,12}\w+\s\w+\|(?P<Worker_ID>\w+\-\w+)\s" | rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task1_State>\w+)\|" | rex field=_raw "(\w+\-){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task1_State>\w+)\|" | rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|(?P<Worker1_ID>\w+\-\w+)\s" | rex field=_raw "(\w+\-){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|(?P<Worker1_ID>\w+\-\w+)\s" | replace "mwgcb-csrla01u_XX_" with "mwgcb-csrla01u" in Worker1_ID | replace "mwgcb-csrla02u_XX_" with "mwgcb-csrla02u" in Worker1_ID | rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task2_State>\w+)" | rex field=_raw "(\w+\-){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task2_State>\w+)" | replace "NA" with "Not_Available" in Task2_State | rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|(?P<Worker2_ID>\w+\-\w+)" | rex field=_raw "(\w+\-){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|(?P<Worker2_ID>\w+\-\w+)" | replace "mwgcb-csrla01u_XX_" with "mwgcb-csrla01u" in Worker2_ID | replace "mwgcb-csrla02u_XX_" with "mwgcb-csrla02u" in Worker2_ID | fillnull value="Not_Available" Task1_State,Worker1_ID,Task2_State,Worker2_ID Second set:
Log1: }
Log2: "type": "sink"
Log3: "tasks": [],
Log4: "name": "sink.http.apac.hk.dna.eap.adobedmp.derived.int.aamrtevents",
Log5: },
Log6: "worker_id": "mwgcb-csrla02u.nam.nsroot.net:8074"
Log7: "trace": "org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed\nCaused by: javax.net.ssl.SSLProtocolException: Unexpected handshake message: server_hello\n\tat sun.security.ssl.Alert.createSSLException(Alert.java:129)\n\tat sun.security.ssl.Alert.createSSLException(Alert.java:117)\n\tat sun.security.ssl.TransportContext.fatal(TransportContext.java:356)\n\tat sun.security.ssl.TransportContext.fatal(TransportContext.java:312)\n\tat sun.security.ssl.TransportContext.fatal(TransportContext.java:303)\n\tat sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:473)\n\tat sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:990)\n\tat sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:977)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:924)\n\tat org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:501)\n\tat org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:593)\n\tat org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:439)\n\tat org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:324)\n\tat org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:205)\n\tat org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:561)\n\tat org.apache.kafka.common.network.Selector.poll(Selector.java:498)\n\tat org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:575)\n\tat org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1358)\n\tat org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1289)\n\tat java.lang.Thread.run(Thread.java:750)\n",
Log8: "state": "FAILED",
Log9: "connector": {
Log10: {
And then the set repeats for the next Connector.
I tried to extract the required fields using below queries:
Query1: ... | regex _raw="name" | rex field=_raw "name\"\:\s\"(?P<Connector>[^\"]+)\"\,"
Query2: ... | rex field=_raw "Caused\sby\:\s(?P<Exception>[^\:]+)\:\s" | search Exception!="org.apache.kafka.common.KafkaException" | rex field=_raw "Caused\sby\:\s([^\:]+)\:\s(?P<Error_Msg>(\w+\:\s){0,1}(\w+\s){1,15}\w+)"
Note that, both the set of logs have different source but same index.
I want to create a table using both the set of logs such that along with the failed connector we can see the error details also.
For e.g.
Connector_Name
Connector_State
Worker_ID
Task1_State
Worker1_ID
Task2_State
Worker2_ID
Exception
Error_Msg
sink.http.apac.hk.dna.eap.adobedmp.derived.int.aamrtevents
FAILED
mwgcb-csrla02u
RUNNING
mwgcb-csrla02u
NA
NA
javax.net.ssl.SSLProtocolException
Unexpected handshake message
sink.http.apac.au.dna.eap.adobedmp.derived.int.aamrtevents
RUNNING
mwgcb-csrla02u
RUNNING
mwgcb-csrla02u
NA
NA
NA
NA
I need to Connector name from second set and compare with first set, and whichever connector has failed state, I need to add the Exception and Error_Msg there.
Please help me to get my desired table. Your kind help is highly appreciated.
Thank you..!!