Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- About Mrig342
Mrig342
Contributor
Member since:
01-26-2021
Online
Community Statistics
| Posts | 106 |
| Solutions | 1 |
| Karma Given | 55 |
| Karma Received | 2 |
| Member Since | 01-26-2021 |
Activity Feed
- Posted Re: Need to create a common table with 2 set of logs on Dashboards & Visualizations. yesterday
- Karma Re: Need to create a common table with 2 set of logs for ITWhisperer. yesterday
- Posted Re: Need to create a common table with 2 set of logs on Dashboards & Visualizations. Thursday
- Karma Re: Need to create a common table with 2 set of logs for ITWhisperer. Thursday
- Posted Re: Need to create a common table with 2 set of logs on Dashboards & Visualizations. Thursday
- Karma Re: Need to create a common table with 2 set of logs for ITWhisperer. Thursday
- Posted Re: Need to create a common table with 2 set of logs on Dashboards & Visualizations. Wednesday
- Karma Re: Need to create a common table with 2 set of logs for ITWhisperer. Wednesday
- Posted Re: Need to create a common table with 2 set of logs on Dashboards & Visualizations. Wednesday
- Karma Re: Need to create a common table with 2 set of logs for ITWhisperer. Wednesday
- Posted Re: Need to create a common table with 2 set of logs on Dashboards & Visualizations. Wednesday
- Posted How to create a common table with two set of logs? on Dashboards & Visualizations. a week ago
- Posted Re: How to combine two set of logs and create a table out of it? on Dashboards & Visualizations. 09-08-2022 10:25 AM
- Posted Re: How to combine two set of logs and create a table out of it? on Dashboards & Visualizations. 09-07-2022 09:53 AM
- Posted Re: How to combine two set of logs and create a table out of it? on Dashboards & Visualizations. 09-06-2022 10:26 AM
- Posted Re: How to combine two set of logs and create a table out of it? on Dashboards & Visualizations. 09-06-2022 05:00 AM
- Karma Re: How to combine two set of logs and create a table out of it? for ITWhisperer. 09-06-2022 05:00 AM
- Posted Re: How to combine two set of logs and create a table out of it? on Dashboards & Visualizations. 09-05-2022 10:15 PM
- Posted How to combine two set of logs and create a table out of it? on Dashboards & Visualizations. 09-05-2022 10:10 AM
- Posted Re: Need help to crate query to pull values from multiple lines on Dashboards & Visualizations. 07-26-2022 01:38 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 |
yesterday
Hi @ITWhisperer, Thank you for all the inputs you provided..!! I am now able to get the table as per my requirement. Your valuable inputs are much appreciated.
... View more
Thursday
Hi @ITWhisperer, I tried removing Time_Stamp from by clause, but this has put forward the Component requirement. The table now has the time_stamp and fills the exception & error_msg columns. But the table now shows all the components instead of having one component at a time. This is the query I used: | eval Time_Stamp=strftime(_time, "%b %d, %Y %I:%M:%S %p")
| eval Component=if(source=="/tmp/temp_connector1.txt",Component,null())
| eval Task1_State=if(source=="/tmp/temp_connector1.txt",Task1_State,null())
| eval Task2_State=if(source=="/tmp/temp_connector1.txt",Task2_State,null())
| eval Time_Stamp=if(source=="/tmp/temp_connector1.txt",Time_Stamp,null())
| where source=="/kfkapps/Kafka/scripts/final_conn_status.txt" OR Component="http" OR Component="mq" OR Component="solace"
| stats values(*) as * by Connector_Name Conn_State
| where source=="/kfkapps/Kafka/scripts/final_conn_status.txt" OR Component="mq"
| table Time_Stamp,Connector_Name,Port,Connector_State,Task1_State,Task2_State,Exception,Error_Msg
| mvexpand Time_Stamp
| dedup Connector_Name
| fillnull value=Not_Available Exception,Error_Msg And this is the table it produces: Time_Stamp Connector_Name Connector_State Task1_State Task2_State Exception Error_Msg Jan 24, 2023 11:46:11 PM sink.mq.apac.hk.dna.eap.adobedmp.derived.int . aamrtevents RUNNING|mwgcb-csrla01u RUNNING|mwgcb-csrla01u RUNNING|mwgcb-csrla01u Not_Available Not_Available Jan 24, 2023 11:46:10 PM sink.http.apac.hk.dna.eap.adobedmp.derived.int . aamrtevents FAILED|s wgcb - csrla02u RUNNING| mwgcb - csrla02u Not_Available SslAuthenticationException SSL handshake failed Jan 24, 2023 11:46:07 PM sink.solace.apac.hk.dna.eap.adobedmp.derived.int . aamrtevents RUNNING|mwgcb-csrla01u RUNNING|mwgcb-csrla02u Not_Available Not_Available Not_Available Can you please help me modify the query to meet both the time-stamp and component requirements..!! Thank you..!!
... View more
Thursday
Hi @ITWhisperer, Thank you for your inputs..!! I am able to get the table for one component at a time now. However, the time requirement is still not resolved. Using the below query: | eval Time_Stamp=strftime(_time, "%b %d, %Y %I:%M:%S %p")
| eval Component=if(source==1,Component,null())
| eval Task1_State=if(source==1,Task1_State,null())
| eval Task2_State=if(source==1,Task2_State,null())
| eval Time_Stamp=if(source==1,Time_Stamp,null())
| where source==2 OR Component="http" OR Component="mq" OR Component="solace"
| stats values(*) as * by Connector_Name Conn_State Time_Stamp
| where source==2 OR Component="mq"
| table Time_Stamp,Connector_Name,Connector_State,Task1_State,Task2_State,Exception,Error_Msg And getting this below table: Time_Stamp Connector_Name Connector_State Task1_State Task2_State Exception Error_Msg Jan 24, 2023 01:50:06 PM sink.http.apac.au.digital.mbol.mbk.raw.int.rawevent FAILED|mwgcb-csrla02u RUNNING|mwgcb-csrla01u Not_Available Not_Available Not_Available Jan 24, 2023 01:50:05 PM sink.http.apac.au.dna.eap.ec.derived.int.amplifycarddetails RUNNING|mwgcb-csrla01u RUNNING|mwgcb-csrla01u RUNNING|mwgcb-csrla01u Not_Available Not_Available As you can see, in the table the Exception & Error_Msg column is filled blank/Not_Available which should have been filled otherwise. Please help on the Time_Stamp requirement to get the table in a meaningful way. Thank You..!!
... View more
Wednesday
Hi @ITWhisperer, Thank you for your inputs. The added query line (| where source==2 OR Component="http" OR Component="mq" OR Component="solace") is giving the table all filled for all type of components in a single table. However, I need the table to be showing one component at a time. For e.g. the table for "mq" component should have only the connectors with mq component. I need to add a query line (| search Component="mq") so that it can be used for drilldown also. And, for the Time_Stamp requirement, I want to consider the time vale from source 1. Please help on the Component & Time_Stamp requirement to get the table in the desired way. Thank You..!!
... View more
Wednesday
Hi @ITWhisperer, Thank you for your inputs. I am getting a table using the query you provided. However, rows are getting blank for components other than "http" I am having Connectors with different Component types. e.g. 1. sink.http.apac.hk.dna.eap.adobedmp.derived.int.aamrtevents 2. sink.mq.apac.hk.dna.eap.adobedmp.derived.int.aamrtevents 3. sink.solace.apac.hk.dna.eap.adobedmp.derived.int.aamrtevents if I am having these connectors in failed state, then I am getting below blanked table: Connector_Name Connector_State Task1_State Task2_State Exception Error_Msg sink.mq.apac.hk.dna.eap.adobedmp.derived.int . aamrtevents Not_Available Not_Available sink.http.apac.hk.dna.eap.adobedmp.derived.int . aamrtevents FAILED|s wgcb - csrla02u RUNNING| mwgcb - csrla02u Not_Available SslAuthenticationException SSL handshake failed sink.solace.apac.hk.dna.eap.adobedmp.derived.int . aamrtevents Not_Available Not_Available I added these extra lines to your query to get the above table: | rex field=_raw "(\w+\.)(?P<Component>\w+)\." | rex field=_raw "(?P<Connector_Name>(\w+\.){3,12}\w+)\s" | rex field=_raw "(?P<Connector_Name>(\w+\-){3,12}\w+)\s" | rex field=_raw "(\w+\.){3,12}\w+\s(?P<Conn_State>\w+)\|" | rex field=_raw "(\w+\.){3,12}\w+\s(?P<Connector_State>\w+\|[^\s]+)\s" | rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task1_State>\w+\|[^\s]+)\s" | replace "RUNNING|mwgcb-csrla01u_XX_" with "RUNNING|mwgcb-csrla01u" in Task1_State | replace "RUNNING|mwgcb-csrla02u_XX_" with "RUNNING|mwgcb-csrla02u" in Task1_State | replace "FAILED|mwgcb-csrla01u_XX_" with "FAILED|mwgcb-csrla01u" in Task1_State | replace "FAILED|mwgcb-csrla02u_XX_" with "FAILED|mwgcb-csrla02u" in Task1_State | rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task2_State>\w+\|[^\s]+)" | replace "RUNNING|mwgcb-csrla01u_XX_" with "RUNNING|mwgcb-csrla01u" in Task2_State | replace "RUNNING|mwgcb-csrla02u_XX_" with "RUNNING|mwgcb-csrla02u" in Task2_State | replace "FAILED|mwgcb-csrla01u_XX_" with "FAILED|mwgcb-csrla01u" in Task2_State | replace "FAILED|mwgcb-csrla02u_XX_" with "FAILED|mwgcb-csrla02u" in Task2_State | fillnull value="Not_Available" Task1_State,Task2_State
| rex field=_raw "\"name\"\:\s\"(?P<Connector_Name>(\w+\.){1,12}\w+)\"\," | rex field=_raw "\{\s+\"state\"\:\s\"(?P<Conn_State>\w+)\"\," | rex field=_raw "\{\s+\"state\"\:\s\"\w+\"\,\s+\"trace\"\:\s\"(\w+\.){1,6}(?P<Exception>\w+)\:\s" | rex field=_raw "\{\s+\"state\"\:\s\"\w+\"\,\s+\"trace\"\:\s\"(\w+\.)+\w+\:\s(?P<Error_Msg>(\w+\s){1,15}\w+)"
| eval Component=if(source==1,Component,null())
| eval Task1_State=if(source==1,Task1_State,null())
| eval Task2_State=if(source==1,Task2_State,null())
| where source==2 OR Component="http"
| stats values(*) as * by Connector_Name Conn_State
| table Connector_Name,Connector_State,Task1_State,Task2_State,Exception,Error_Msg
| dedup Connector_Name
| fillnull value=Not_Available Exception,Error_Msg When I change the line (| where source==2 OR Component="http") to (| where source==2 OR Component="http" OR Component="mq" OR Component="solace"), I am getting the table properly filled with the data. However, I need the table to be showing one component at a time. I tried adding the query (| search Component="solace"), but it makes the Exception & Error_Msg fields empty. Similarly, when trying to add a Time_stamp column to the table, it makes the Exception & Error_Msg fields empty again. For adding time stamp, I added these lines of query: | rex field=_raw "(\w+\.)(?P<Component>\w+)\." | rex field=_raw "(?P<Connector_Name>(\w+\.){3,12}\w+)\s" | rex field=_raw "(?P<Connector_Name>(\w+\-){3,12}\w+)\s" | rex field=_raw "(\w+\.){3,12}\w+\s(?P<Conn_State>\w+)\|" | rex field=_raw "(\w+\.){3,12}\w+\s(?P<Connector_State>\w+\|[^\s]+)\s" | rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task1_State>\w+\|[^\s]+)\s" | replace "RUNNING|mwgcb-csrla01u_XX_" with "RUNNING|mwgcb-csrla01u" in Task1_State | replace "RUNNING|mwgcb-csrla02u_XX_" with "RUNNING|mwgcb-csrla02u" in Task1_State | replace "FAILED|mwgcb-csrla01u_XX_" with "FAILED|mwgcb-csrla01u" in Task1_State | replace "FAILED|mwgcb-csrla02u_XX_" with "FAILED|mwgcb-csrla02u" in Task1_State | rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task2_State>\w+\|[^\s]+)" | replace "RUNNING|mwgcb-csrla01u_XX_" with "RUNNING|mwgcb-csrla01u" in Task2_State | replace "RUNNING|mwgcb-csrla02u_XX_" with "RUNNING|mwgcb-csrla02u" in Task2_State | replace "FAILED|mwgcb-csrla01u_XX_" with "FAILED|mwgcb-csrla01u" in Task2_State | replace "FAILED|mwgcb-csrla02u_XX_" with "FAILED|mwgcb-csrla02u" in Task2_State | fillnull value="Not_Available" Task1_State,Task2_State
| rex field=_raw "\"name\"\:\s\"(?P<Connector_Name>(\w+\.){1,12}\w+)\"\," | rex field=_raw "\{\s+\"state\"\:\s\"(?P<Conn_State>\w+)\"\," | rex field=_raw "\{\s+\"state\"\:\s\"\w+\"\,\s+\"trace\"\:\s\"(\w+\.){1,6}(?P<Exception>\w+)\:\s" | rex field=_raw "\{\s+\"state\"\:\s\"\w+\"\,\s+\"trace\"\:\s\"(\w+\.)+\w+\:\s(?P<Error_Msg>(\w+\s){1,15}\w+)"
| eval Time_Stamp=strftime(_time, "%b %d, %Y %I:%M:%S %p")
| eval Component=if(source==1,Component,null())
| eval Task1_State=if(source==1,Task1_State,null())
| eval Task2_State=if(source==1,Task2_State,null())
| where source==2 OR Component="http"
| stats values(*) as * by Connector_Name Conn_State Time_Stamp
| table Time_Stamp,Connector_Name,Connector_State,Task1_State,Task2_State,Exception,Error_Msg
| dedup Connector_Name
| fillnull value=Not_Available Exception,Error_Msg Please help on the Component & Time_Stamp requirement to get the table in a meaningful way. Thank You..!!
... View more
Wednesday
Hi @ITWhisperer, I made some changes to the input logs. Please see below and help to get desired table. First set:
Log1: sink.http.apac.hk.dna.eap.adobedmp.derived.int.aamrtevents FAILED|swgcb-csrla02u RUNNING|mwgcb-csrla02u NA
Log2: sink.http.apac.au.dna.eap.adobedmp.derived.int.aamrtevents RUNNING|mwgcb-csrla01u RUNNING|mwgcb-csrla01u NA
Log3: sink.http.apac.ph.dna.eap.adobedmp.derived.int.aamrtevents FAILED|mwgcb-csrla01u FAILED|mwgcb-csrla01u NA
Log4: sink.http.apac.th.dna.eap.adobedmp.derived.int.aamrtevents FAILED|swgcb-csrla01u FAILED|mwgcb-csrla01u NA
Here I used the below query to extract the required fields:
... | rex field=_raw "(\w+\.)(?P<Component>\w+)\." | search Component=http | rex field=_raw "(?P<Connector_Name>(\w+\.){3,12}\w+)\s" | rex field=_raw "(?P<Connector_Name>(\w+\-){3,12}\w+)\s" | rex field=_raw "(\w+\.){3,12}\w+\s(?P<Conn_State>\w+)\|" | rex field=_raw "(\w+\.){3,12}\w+\s(?P<Connector_State>\w+\|[^\s]+)\s" | rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task1_State>\w+\|[^\s]+)\s" | replace "RUNNING|mwgcb-csrla01u_XX_" with "RUNNING|mwgcb-csrla01u" in Task1_State | replace "RUNNING|mwgcb-csrla02u_XX_" with "RUNNING|mwgcb-csrla02u" in Task1_State | replace "FAILED|mwgcb-csrla01u_XX_" with "FAILED|mwgcb-csrla01u" in Task1_State | replace "FAILED|mwgcb-csrla02u_XX_" with "FAILED|mwgcb-csrla02u" in Task1_State | rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task2_State>\w+\|[^\s]+)" | replace "RUNNING|mwgcb-csrla01u_XX_" with "RUNNING|mwgcb-csrla01u" in Task2_State | replace "RUNNING|mwgcb-csrla02u_XX_" with "RUNNING|mwgcb-csrla02u" in Task2_State | replace "FAILED|mwgcb-csrla01u_XX_" with "FAILED|mwgcb-csrla01u" in Task2_State | replace "FAILED|mwgcb-csrla02u_XX_" with "FAILED|mwgcb-csrla02u" in Task2_State | fillnull value="Not_Available" Task1_State,Task2_State Second set:
Log1: {
"connector": {
"state": "RUNNING",
"worker_id": "mwgcb-csrla01u.nam.nsroot.net:8074"
},
"name": "sink.http.apac.au.dna.eap.adobedmp.derived.int.aamrtevents",
"tasks": [
{
"id": 0,
"state": "RUNNING",
"worker_id": "mwgcb-csrla01u.nam.nsroot.net:8074"
}
],
"type": "sink"
}
Log2:
{
"connector": {
"state": "FAILED",
"trace": "org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed\nCaused by: javax.net.ssl.SSLProtocolException: Unexpected handshake message: server_hello\n\tat sun.security.ssl.Alert.createSSLException(Alert.java:129)\n\tat sun.security.ssl.Alert.createSSLException(Alert.java:117)\n\tat sun.security.ssl.TransportContext.fatal(TransportContext.java:356)\n\tat sun.security.ssl.TransportContext.fatal(TransportContext.java:312)\n\tat sun.security.ssl.TransportContext.fatal(TransportContext.java:303)\n\tat sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:473)\n\tat sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:990)\n\tat sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:977)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:924)\n\tat org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:501)\n\tat org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:593)\n\tat org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:439)\n\tat org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:324)\n\tat org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:205)\n\tat org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:561)\n\tat org.apache.kafka.common.network.Selector.poll(Selector.java:498)\n\tat org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:575)\n\tat org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1358)\n\tat org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1289)\n\tat java.lang.Thread.run(Thread.java:750)\n",
"worker_id": "swgcb-csrla02u.nam.nsroot.net:8098"
},
"name": "sink.http.apac.hk.dna.eap.adobedmp.derived.int.aamrtevents",
"tasks": [
{
"id": 0,
"state": "RUNNING",
"worker_id": "mwgcb-csrla02u.nam.nsroot.net:8098"
}
],
"type": "source"
}
Here I used the below query to extract the required fields:
... | rex field=_raw "\"name\"\:\s\"(?P<Connector_Name>(\w+\.){1,12}\w+)\"\," | rex field=_raw "\{\s+\"state\"\:\s\"(?P<Conn_State>\w+)\"\," | rex field=_raw "\{\s+\"state\"\:\s\"\w+\"\,\s+\"trace\"\:\s\"(\w+\.){1,6}(?P<Exception>\w+)\:\s" | rex field=_raw "\{\s+\"state\"\:\s\"\w+\"\,\s+\"trace\"\:\s\"(\w+\.)+\w+\:\s(?P<Error_Msg>(\w+\s){1,15}\w+)" Note that, both the set of logs have different source but same index. I want to create a table using both the set of logs such that along with the failed connector we can see the error details also. For e.g. Connector_Name Connector_State Task1_State Task2_State Exception Error_Msg sink.http.apac.au.dna.eap.adobedmp.derived.int . aamrtevents RUNNING| mwgcb - csrla01u RUNNING| mwgcb - csrla01u Not_Available Not_Available Not_Available sink.http.apac.hk.dna.eap.adobedmp.derived.int . aamrtevents FAILED|s wgcb - csrla02u RUNNING| mwgcb - csrla02u Not_Available SslAuthenticationException SSL handshake failed I need Connector_Name from second set tocompare with Connector_Name from first set, and whichever connector has failed state, I need to add the Exception and Error_Msg there. Please help me to get my desired table. Your kind help is highly appreciated. Thank You..!!
... View more
a week ago
Hi All,
I have got 2 set of logs, one of which has the Connector details and the other has got the error details if any connector gets failed.
First set:
Log1: sink.http.apac.hk.dna.eap.adobedmp.derived.int.aamrtevents FAILED|mwgcb-csrla02u RUNNING|mwgcb-csrla02u NA
Log2: sink.http.apac.au.dna.eap.adobedmp.derived.int.aamrtevents RUNNING|mwgcb-csrla02u RUNNING|mwgcb-csrla02u NA
Log3: sink.http.apac.ph.dna.eap.adobedmp.derived.int.aamrtevents FAILED|mwgcb-csrla01u FAILED|mwgcb-csrla01u NA
Log4: sink.http.apac.th.dna.eap.adobedmp.derived.int.aamrtevents FAILED|swgcb-csrla01u FAILED|mwgcb-csrla01u NA
Here I used the below query to extract the required fields:
... | rex field=_raw "(\w+\.)(?P<Component>\w+)\." | rex field=_raw "(?P<Connector_Name>(\w+\.){3,12}\w+)\s" | rex field=_raw "(?P<Connector_Name>(\w+\-){3,12}\w+)\s" | rex field=_raw "(\w+\.){3,12}\w+\s(?P<Connector_State>\w+)\|" | rex field=_raw "(\w+\-){3,12}\w+\s(?P<Connector_State>\w+)\|" | rex field=_raw "(\w+\.){3,12}\w+\s\w+\|(?P<Worker_ID>\w+\-\w+)\s" | rex field=_raw "(\w+\-){3,12}\w+\s\w+\|(?P<Worker_ID>\w+\-\w+)\s" | rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task1_State>\w+)\|" | rex field=_raw "(\w+\-){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task1_State>\w+)\|" | rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|(?P<Worker1_ID>\w+\-\w+)\s" | rex field=_raw "(\w+\-){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|(?P<Worker1_ID>\w+\-\w+)\s" | replace "mwgcb-csrla01u_XX_" with "mwgcb-csrla01u" in Worker1_ID | replace "mwgcb-csrla02u_XX_" with "mwgcb-csrla02u" in Worker1_ID | rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task2_State>\w+)" | rex field=_raw "(\w+\-){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task2_State>\w+)" | replace "NA" with "Not_Available" in Task2_State | rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|(?P<Worker2_ID>\w+\-\w+)" | rex field=_raw "(\w+\-){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|(?P<Worker2_ID>\w+\-\w+)" | replace "mwgcb-csrla01u_XX_" with "mwgcb-csrla01u" in Worker2_ID | replace "mwgcb-csrla02u_XX_" with "mwgcb-csrla02u" in Worker2_ID | fillnull value="Not_Available" Task1_State,Worker1_ID,Task2_State,Worker2_ID Second set:
Log1: }
Log2: "type": "sink"
Log3: "tasks": [],
Log4: "name": "sink.http.apac.hk.dna.eap.adobedmp.derived.int.aamrtevents",
Log5: },
Log6: "worker_id": "mwgcb-csrla02u.nam.nsroot.net:8074"
Log7: "trace": "org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed\nCaused by: javax.net.ssl.SSLProtocolException: Unexpected handshake message: server_hello\n\tat sun.security.ssl.Alert.createSSLException(Alert.java:129)\n\tat sun.security.ssl.Alert.createSSLException(Alert.java:117)\n\tat sun.security.ssl.TransportContext.fatal(TransportContext.java:356)\n\tat sun.security.ssl.TransportContext.fatal(TransportContext.java:312)\n\tat sun.security.ssl.TransportContext.fatal(TransportContext.java:303)\n\tat sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:473)\n\tat sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:990)\n\tat sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:977)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:924)\n\tat org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:501)\n\tat org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:593)\n\tat org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:439)\n\tat org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:324)\n\tat org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:205)\n\tat org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:561)\n\tat org.apache.kafka.common.network.Selector.poll(Selector.java:498)\n\tat org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:575)\n\tat org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1358)\n\tat org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1289)\n\tat java.lang.Thread.run(Thread.java:750)\n",
Log8: "state": "FAILED",
Log9: "connector": {
Log10: {
And then the set repeats for the next Connector.
I tried to extract the required fields using below queries:
Query1: ... | regex _raw="name" | rex field=_raw "name\"\:\s\"(?P<Connector>[^\"]+)\"\,"
Query2: ... | rex field=_raw "Caused\sby\:\s(?P<Exception>[^\:]+)\:\s" | search Exception!="org.apache.kafka.common.KafkaException" | rex field=_raw "Caused\sby\:\s([^\:]+)\:\s(?P<Error_Msg>(\w+\:\s){0,1}(\w+\s){1,15}\w+)"
Note that, both the set of logs have different source but same index.
I want to create a table using both the set of logs such that along with the failed connector we can see the error details also.
For e.g.
Connector_Name
Connector_State
Worker_ID
Task1_State
Worker1_ID
Task2_State
Worker2_ID
Exception
Error_Msg
sink.http.apac.hk.dna.eap.adobedmp.derived.int . aamrtevents
FAILED
mwgcb - csrla02u
RUNNING
mwgcb - csrla02u
NA
NA
javax.net.ssl.SSLProtocolException
Unexpected handshake message
sink.http.apac.au.dna.eap.adobedmp.derived.int . aamrtevents
RUNNING
mwgcb - csrla02u
RUNNING
mwgcb - csrla02u
NA
NA
NA
NA
I need to Connector name from second set and compare with first set, and whichever connector has failed state, I need to add the Exception and Error_Msg there.
Please help me to get my desired table. Your kind help is highly appreciated.
Thank you..!!
... View more
Labels
- Labels:
-
table
09-08-2022
10:25 AM
Hi @ITWhisperer We have the connector count fixed for each node and in the search I am using a lookup table to identify which connector is for which node.
... View more
09-07-2022
09:53 AM
Hi @ITWhisperer First of all both the log types are from the same index, host and source and if I search "search index=ABC host=XYZ source=KLM" I get below data: Log1: MACHINE@|@Port@|@Country@|@Count MEMORY STATUS
mwgcb-csrla01u|8070|EAS|5 CNF_| PASS|mwgcb-csrla01u PASS|mwgcb-csrla02u
Log2: source.mq.apac.sg.cards.eas.eas.raw.int.rawevent RUNNING|mwgcb-csrla02u RUNNING|mwgcb-csrla01u RUNNING|mwgcb-csrla02u
Log3: source.mq.apac.in.cards.eas.eas.raw.int.rawevent RUNNING|mwgcb-csrla01u FAILED|mwgcb-csrla02u NA
Log4: source.mq.apac.my.cards.eas.eas.raw.int.rawevent FAILED|mwgcb-csrla02u RUNNING|mwgcb-csrla01u NA
Log5: source.mq.apac.th.cards.eas.eas.raw.int.rawevent RUNNING|mwgcb-csrla01u RUNNING|mwgcb-csrla01u NA
Log6: source.mq.apac.hk.cards.eas.eas.raw.int.rawevent UNASSIGNED|mwgcb-csrla01u RUNNING|mwgcb-csrla01u RUNNING|mwgcb-csrla02u
Log7: MACHINE@|@Port@|@Country@|@Count MEMORY STATUS
mwgcb-csrla01u|8070|ESB|6 CNF_| PASS|mwgcb-csrla01u PASS|mwgcb-csrla02u
Log8: source.mq.apac.sg.cards.esb.esb.raw.int.rawevent RUNNING|mwgcb-csrla02u RUNNING|mwgcb-csrla01u RUNNING|mwgcb-csrla02u
Log9: source.mq.apac.in.cards.esb.esb.raw.int.rawevent RUNNING|mwgcb-csrla01u FAILED|mwgcb-csrla02u NA
Log10: source.mq.apac.my.cards.esb.esb.raw.int.rawevent FAILED|mwgcb-csrla02u RUNNING|mwgcb-csrla01u NA
Log11: source.mq.apac.th.cards.esb.esb.raw.int.rawevent RUNNING|mwgcb-csrla01u RUNNING|mwgcb-csrla01u NA
Log12: source.mq.apac.hk.cards.esb.esb.raw.int.rawevent UNASSIGNED|mwgcb-csrla01u RUNNING|mwgcb-csrla01u RUNNING|mwgcb-csrla02u
Log13: source.solace.apac.cn.digital.esb.esb.derived.int.esblogs RUNNING|mwgcb-csrla02u RUNNING|mwgcb-csrla01u NA The first set of logs are giving the Count of Connectors available in each worker Node. And the second set is giving the Connector states from which I want to count the Count the connectors that are Running i.e. successful and put all together in a table in below manner. Machine_Name Port Worker_Node Connector_Count Success_Count mwgcb - csrla01u 8070 EAS 5 3 mwgcb - csrla01u 8070 ESB 6 4 I hope this info helps. Please help me to create the table in the desired manner.
... View more
09-06-2022
10:26 AM
Hi @ITWhisperer For the above result, the full query was: | multisearch [
search index=ABC host=XYZ source=KLM
| regex _raw="\w+\-\w+\|\d+"
| rex field=_raw "(?P<Machine_Name>\w+\-\w+)\|(?P<Port>\d+)\|(?P<Worker_Node>\w+)\|(?P<Connector_Count>\d+)\s"
]
[search index=ABC host=XYZ source=KLM | regex _raw!="\w+\-\w+\|\d+"
| regex _raw!="properties"
| regex _raw!="MACHINE"
| regex _raw!="CONNECTOR_NAME"
| regex _raw!="CNF"
| regex _raw!="Detailed"
| rex field=_raw "(?P<Connector_Name>(\w+\.){3,12}\w+)\s"
| rex field=_raw "(?P<Connector_Name>(\w+\-){3,12}\w+)\s"
| rex field=_raw "(\w+\.){3,12}\w+\s(?P<Connector_State>\w+)\|"
| rex field=_raw "(\w+\-){3,12}\w+\s(?P<Connector_State>\w+)\|"
| rex field=_raw "(\w+\.){3,12}\w+\s\w+\|(?P<Worker_ID>\w+\-\w+)\s"
| rex field=_raw "(\w+\-){3,12}\w+\s\w+\|(?P<Worker_ID>\w+\-\w+)\s"
| rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task1_State>\w+)\|"
| rex field=_raw "(\w+\-){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task1_State>\w+)\|"
| rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|(?P<Worker1_ID>\w+\-\w+)\s"
| rex field=_raw "(\w+\-){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|(?P<Worker1_ID>\w+\-\w+)\s"
| replace "mwgcb-csrla01u_XX_" with "mwgcb-csrla01u" in Worker1_ID
| replace "mwgcb-csrla02u_XX_" with "mwgcb-csrla02u" in Worker1_ID
| rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task2_State>\w+)"
| rex field=_raw "(\w+\-){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task2_State>\w+)"
| replace "NA" with "Not_Available" in Task2_State
| rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|(?P<Worker2_ID>\w+\-\w+)"
| rex field=_raw "(\w+\-){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|(?P<Worker2_ID>\w+\-\w+)"
| replace "mwgcb-csrla01u_XX_" with "mwgcb-csrla01u" in Worker2_ID
| replace "mwgcb-csrla02u_XX_" with "mwgcb-csrla02u" in Worker2_ID
| fillnull value="Not_Available" Task1_State, Worker1_ID, Task2_State, Worker2_ID
]
| lookup Worker_Connector_List.csv "Connector_Name"
| search Worker_Node=EAS | stats values(Machine_Name) as Machine_Name, values(Port) as Port, values(Worker_Node) as Worker_Node, values(Connector_Count) as Connector_Count, latest(Connector_State) as Connector_State by Connector_Name | stats count(eval(Connector_State="RUNNING")) as Success_Count | table Machine_Name,Port,Worker_Node,Connector_Count,Success_Count While I used the query you provide, the full search was: | multisearch [
search index=ABC host=XYZ source=KLM
| regex _raw="\w+\-\w+\|\d+"
| rex field=_raw "(?P<Machine_Name>\w+\-\w+)\|(?P<Port>\d+)\|(?P<Worker_Node>\w+)\|(?P<Connector_Count>\d+)\s"
]
[search index=ABC host=XYZ source=KLM | regex _raw!="\w+\-\w+\|\d+"
| regex _raw!="properties"
| regex _raw!="MACHINE"
| regex _raw!="CONNECTOR_NAME"
| regex _raw!="CNF"
| regex _raw!="Detailed"
| rex field=_raw "(?P<Connector_Name>(\w+\.){3,12}\w+)\s"
| rex field=_raw "(?P<Connector_Name>(\w+\-){3,12}\w+)\s"
| rex field=_raw "(\w+\.){3,12}\w+\s(?P<Connector_State>\w+)\|"
| rex field=_raw "(\w+\-){3,12}\w+\s(?P<Connector_State>\w+)\|"
| rex field=_raw "(\w+\.){3,12}\w+\s\w+\|(?P<Worker_ID>\w+\-\w+)\s"
| rex field=_raw "(\w+\-){3,12}\w+\s\w+\|(?P<Worker_ID>\w+\-\w+)\s"
| rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task1_State>\w+)\|"
| rex field=_raw "(\w+\-){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task1_State>\w+)\|"
| rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|(?P<Worker1_ID>\w+\-\w+)\s"
| rex field=_raw "(\w+\-){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|(?P<Worker1_ID>\w+\-\w+)\s"
| replace "mwgcb-csrla01u_XX_" with "mwgcb-csrla01u" in Worker1_ID
| replace "mwgcb-csrla02u_XX_" with "mwgcb-csrla02u" in Worker1_ID
| rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task2_State>\w+)"
| rex field=_raw "(\w+\-){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task2_State>\w+)"
| replace "NA" with "Not_Available" in Task2_State
| rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|(?P<Worker2_ID>\w+\-\w+)"
| rex field=_raw "(\w+\-){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|(?P<Worker2_ID>\w+\-\w+)"
| replace "mwgcb-csrla01u_XX_" with "mwgcb-csrla01u" in Worker2_ID
| replace "mwgcb-csrla02u_XX_" with "mwgcb-csrla02u" in Worker2_ID
| fillnull value="Not_Available" Task1_State, Worker1_ID, Task2_State, Worker2_ID
]
| lookup Worker_Connector_List.csv "Connector_Name"
| search Worker_Node=EAS | stats values(Machine_Name) as Machine_Name, values(Port) as Port, values(Worker_Node) as Worker_Node, values(Connector_Count) as Connector_Count, latest(Connector_State) as Connector_State, count(eval(Connector_State="RUNNING")) as Success_Count by Connector_Name
| table Machine_Name,Port,Worker_Node,Connector_Count,Success_Count And it gave the below table: Machine_Name Port Worker_Node Connector_Count Success_Count EAS 1 EAS 1 EAS 0 EAS 1 EAS 0 This is not expected table. The table should have one row for each Worker_Node instead of one row of each Connector_Name.
... View more
09-06-2022
05:00 AM
Thank you @ITWhisperer ...!! But it didn't help create the table in the expected manner. Upon using the given query the table came out as below: Machine_Name Port Worker_Node Connector_Count Success_Count 3 Please help me to create/modify the script to get the expected table. You kind help is highly appreciated..!!
... View more
09-05-2022
10:15 PM
Hi All, Can anyone please help me on this..?
... View more
09-05-2022
10:10 AM
Hi All,
I have two set of logs as below and I want a create a table combining them.
Type1:
Log1: MACHINE@|@Port@|@Country@|@Count MEMORY STATUS
mwgcb-csrla01u|8070|EAS|5 CNF_| PASS|mwgcb-csrla01u PASS|mwgcb-csrla02u
Type2:
Log1: source.mq.apac.sg.cards.eas.eas.raw.int.rawevent RUNNING|mwgcb-csrla02u RUNNING|mwgcb-csrla01u RUNNING|mwgcb-csrla02u
Log2: source.mq.apac.in.cards.eas.eas.raw.int.rawevent RUNNING|mwgcb-csrla01u FAILED|mwgcb-csrla02u NA
Log3: source.mq.apac.my.cards.eas.eas.raw.int.rawevent FAILED|mwgcb-csrla02u RUNNING|mwgcb-csrla01u NA
Log4: source.mq.apac.th.cards.eas.eas.raw.int.rawevent RUNNING|mwgcb-csrla01u RUNNING|mwgcb-csrla01u NA
Log5: source.mq.apac.hk.cards.eas.eas.raw.int.rawevent UNASSIGNED|mwgcb-csrla01u RUNNING|mwgcb-csrla01u RUNNING|mwgcb-csrla02u
I extracted the required fields from each of the log types and am trying to create a table with the fields Machine_Name, Port, Worker_Node, Connector_Count, Success_Count where Success_Count is the number of Connectors that are in RUNNING state for a Worker_Node.
For e.g.: For the above set of logs the table should look like
Machine_Name
Port
Worker_Node
Connector_Count
Success_Count
mwgcb - csrla01u
8070
EAS
5
3
I tried to combine the two set of logs by creating a query as below but not successful in getting the above table.
| multisearch [
search index=ABC host=XYZ source=KLM
| regex _raw="\w+\-\w+\|\d+"
| rex field=_raw "(?P<Machine_Name>\w+\-\w+)\|(?P<Port>\d+)\|(?P<Worker_Node>\w+)\|(?P<Connector_Count>\d+)\s"
]
[search index=ABC host=XYZ source=KLM | regex _raw!="\w+\-\w+\|\d+"
| regex _raw!="properties"
| regex _raw!="MACHINE"
| regex _raw!="CONNECTOR_NAME"
| regex _raw!="CNF"
| regex _raw!="Detailed"
| rex field=_raw "(?P<Connector_Name>(\w+\.){3,12}\w+)\s"
| rex field=_raw "(?P<Connector_Name>(\w+\-){3,12}\w+)\s"
| rex field=_raw "(\w+\.){3,12}\w+\s(?P<Connector_State>\w+)\|"
| rex field=_raw "(\w+\-){3,12}\w+\s(?P<Connector_State>\w+)\|"
| rex field=_raw "(\w+\.){3,12}\w+\s\w+\|(?P<Worker_ID>\w+\-\w+)\s"
| rex field=_raw "(\w+\-){3,12}\w+\s\w+\|(?P<Worker_ID>\w+\-\w+)\s"
| rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task1_State>\w+)\|"
| rex field=_raw "(\w+\-){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task1_State>\w+)\|"
| rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|(?P<Worker1_ID>\w+\-\w+)\s"
| rex field=_raw "(\w+\-){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|(?P<Worker1_ID>\w+\-\w+)\s"
| replace "mwgcb-csrla01u_XX_" with "mwgcb-csrla01u" in Worker1_ID
| replace "mwgcb-csrla02u_XX_" with "mwgcb-csrla02u" in Worker1_ID
| rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task2_State>\w+)"
| rex field=_raw "(\w+\-){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|\w+\-\w+\s((\_KK\_){0,1})(?P<Task2_State>\w+)"
| replace "NA" with "Not_Available" in Task2_State
| rex field=_raw "(\w+\.){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|(?P<Worker2_ID>\w+\-\w+)"
| rex field=_raw "(\w+\-){3,12}\w+\s\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|\w+\-\w+\s((\_KK\_){0,1})\w+\|(?P<Worker2_ID>\w+\-\w+)"
| replace "mwgcb-csrla01u_XX_" with "mwgcb-csrla01u" in Worker2_ID
| replace "mwgcb-csrla02u_XX_" with "mwgcb-csrla02u" in Worker2_ID
| fillnull value="Not_Available" Task1_State, Worker1_ID, Task2_State, Worker2_ID
]
| lookup Worker_Connector_List.csv "Connector_Name"
| search Worker_Node=EAS
| stats latest(Connector_State) as Connector_State by Connector_Name
| eval Status=if(Connector_State="RUNNING", "1","0")
| stats sum(Status) as Success_Count
| table Machine_Name,Port,Worker_Node,Connector_Count,Success_Count
Please help me to create/modify the query so that I can get the table in the desired manner.
Thank you All..!!
... View more
Labels
- Labels:
-
table
07-26-2022
01:38 AM
Hi @JacekF... Thank you very much for your help on the query..!! This modified query is giving me the expected tabular results.
... View more
07-25-2022
05:43 AM
Hi @JacekF... Using max_match=0 didn't work.. I tried using max_match=0 after removing "CLIENT-ID\s" and that didn't work either.. Can you please modify it some other way to get the expected result.. Thank you..!!
... View more
07-25-2022
02:06 AM
Hi All,
I have logs like below and want to create a table out of it.
log1:
"connector": {
"state": "RUNNING",
},
"tasks": [
{
"id": 0,
"state": "RUNNING",
}
],
"type": "sink"
}
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
connect-ABC ABC.sinkevents 0 15087148 15087148 0 connector-consumer-ABC /10.231.95.96 connector-consumer-ABC.sinkevents-0
log2:
"connector": {
"state": "RUNNING",
},
"tasks": [
{
"id": 0,
"state": "FAILED",
}
],
"type": "sink"
}
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
connect-XYZ XYZ.cardtransactionauthorizationalertsent 0 27775 27780 5 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 1 27740 27747 7 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 2 27836 27836 0 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
I created the query which give the below table:
.... | rex field=_raw "CLIENT\-ID\s+(?P<Group>[^\s]+)\s(?P<Topic>[^\s]+)\s(?P<Partition>[^\s]+)\s+(?P<Current_Offset>[^\s]+)\s+(?P<Log_End_Offset>[^\s]+)\s+(?P<Lag>[^\s]+)\s+(?P<Consumer_ID>[^\s]+)\s{0,20}(?P<Host>[^\s]+)\s+(?P<Client_ID>[^\s]+)" | table Group,Topic,Partition,Lag,Consumer_ID
Group
Topic
Partition
Lag
Consumer_ID
connect-ABC
ABC.sinkevents
0
0
connector-consumer-ABC
connect-XYZ
XYZ.cardtransactionauthorizationalertsent
0
5
connector-consumer-XYZ
Here I am missing the last 2 lines of log2. I want to modify the query in a way that it produces the table in below manner:
Group
Topic
Partition
Lag
Consumer_ID
connect-ABC
ABC.sinkevents
0
0
connector-consumer-ABC
connect-XYZ
XYZ.cardtransactionauthorizationalertsent
0
5
connector-consumer-XYZ
connect-XYZ
XYZ.cardtransactionauthorizationalertsent
1
7
connector-consumer-XYZ
connect-XYZ
XYZ.cardtransactionauthorizationalertsent
2
0
connector-consumer-XYZ
Please help me to modify the query in a way to get my desired output.
Your kind help on this is highly appreciated.
Thank You..!!
... View more
Labels
- Labels:
-
table
07-24-2022
10:24 AM
Hi @richgalloway Thank you for your suggestion..!! But it is not giving the expected result. May be this is because I missed to update that sometimes there may be other lines before "GROUP" in the logs. My bad..!! Please consider the logs in this way and help me to create the query. log1:
"connector": {
"state": "RUNNING",
},
"tasks": [
{
"id": 0,
"state": "RUNNING",
}
],
"type": "sink"
}
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
connect-ABC ABC.sinkevents 0 15087148 15087148 0 connector-consumer-ABC /10.231.95.96 connector-consumer-ABC.sinkevents-0
log2:
"connector": {
"state": "RUNNING",
},
"tasks": [
{
"id": 0,
"state": "FAILED",
}
],
"type": "sink"
}
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
connect-XYZ XYZ.cardtransactionauthorizationalertsent 0 27775 27780 5 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 1 27740 27747 7 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 2 27836 27836 0 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0 Thank You..!!
... View more
07-24-2022
05:04 AM
Hi All,
I have logs like below and want to create a table out of it.
log1:
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
connect-ABC ABC.sinkevents 0 15087148 15087148 0 connector-consumer-ABC /10.231.95.96 connector-consumer-ABC.sinkevents-0
log2:
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
connect-XYZ XYZ.cardtransactionauthorizationalertsent 0 27775 27780 5 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 1 27740 27747 7 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 2 27836 27836 0 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
I created the query which give the below table:
.... | rex field=_raw "CLIENT\-ID\s+(?P<Group>[^\s]+)\s(?P<Topic>[^\s]+)\s(?P<Partition>[^\s]+)\s+(?P<Current_Offset>[^\s]+)\s+(?P<Log_End_Offset>[^\s]+)\s+(?P<Lag>[^\s]+)\s+(?P<Consumer_ID>[^\s]+)\s{0,20}(?P<Host>[^\s]+)\s+(?P<Client_ID>[^\s]+)" | table Group,Topic,Partition,Lag,Consumer_ID
Group
Topic
Partition
Lag
Consumer_ID
connect-ABC
ABC.sinkevents
0
0
connector-consumer-ABC
connect-XYZ
XYZ.cardtransactionauthorizationalertsent
0
5
connector-consumer-XYZ
Here I am missing the last 2 lines of log2. I want to modify the query in a way that it produces the table in below manner:
Group
Topic
Partition
Lag
Consumer_ID
connect-ABC
ABC.sinkevents
0
0
connector-consumer-ABC
connect-XYZ
XYZ.cardtransactionauthorizationalertsent
0
5
connector-consumer-XYZ
connect-XYZ
XYZ.cardtransactionauthorizationalertsent
1
7
connector-consumer-XYZ
connect-XYZ
XYZ.cardtransactionauthorizationalertsent
2
0
connector-consumer-XYZ
Please help me to modify the query in a way to get my desired output.
Your kind help on this is highly appreciated.
Thank You..!!
... View more
Labels
- Labels:
-
table
07-05-2022
09:11 AM
1 Karma
Thank you @richgalloway for your insights. I checked for the spaces between the values and they were equally spaced. Still couldn't get the expected result. Then I replaced the normal spaces between the values with tab spaces in the logs. And that gave me the result in the expected manner.
... View more
07-02-2022
09:56 AM
Thank you @richgalloway ...!! The command multikv helped me break the events per line-wise. However when I used table command to create a table for dashboard it is not what I desired. I used this query "**** | multikv forceheader=1 | table ServerA,ServerB,ServerC" and this gave the table as: Server ServerA ServerB ServerC Application ADFILES41-6.2-4 not_available ADFILES41-6.2-4 Application ADM41-5.10.1-4 ADM41-5.10.1-4 ADM41-5.10.1-4 Application ADM41HF-5.10.1HF004-4 ADM41HF-5.10.1HF004-4 ADM41HF-5.10.1HF004-4 Application ADM42-5.11-4 ADM42-5.11-4 ADM42-5.11-4 Application ADM42HF-5.11HF03-4 ADM42HF-5.11HF03-4 not_available Application TRA42-5.11-4 TRA42-5.11-4 not_available Application not_available ADFILES42-6.2-4 not_available Application not_available not_available TRA42-5.13-4 I am not able to understand what went wrong here in the query, as the logs are equally spaced and are in a tabular format. Please help me identify what might have gone wrong or help me modify the query to get the desired output. Any help on this is highly appreciated. Thank You All..!!
... View more
Contact Me
| Online Status |
Online
|
| Date Last Visited |
yesterday
|
Karma given to
