Hi All,
I have logs like below and want to create a table out of it.
log1:
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
connect-ABC ABC.sinkevents 0 15087148 15087148 0 connector-consumer-ABC /10.231.95.96 connector-consumer-ABC.sinkevents-0
log2:
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
connect-XYZ XYZ.cardtransactionauthorizationalertsent 0 27775 27780 5 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 1 27740 27747 7 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 2 27836 27836 0 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
I created the query which give the below table:
.... | rex field=_raw "CLIENT\-ID\s+(?P<Group>[^\s]+)\s(?P<Topic>[^\s]+)\s(?P<Partition>[^\s]+)\s+(?P<Current_Offset>[^\s]+)\s+(?P<Log_End_Offset>[^\s]+)\s+(?P<Lag>[^\s]+)\s+(?P<Consumer_ID>[^\s]+)\s{0,20}(?P<Host>[^\s]+)\s+(?P<Client_ID>[^\s]+)" | table Group,Topic,Partition,Lag,Consumer_ID
Group | Topic | Partition | Lag | Consumer_ID |
connect-ABC | ABC.sinkevents | 0 | 0 | connector-consumer-ABC |
connect-XYZ | XYZ.cardtransactionauthorizationalertsent | 0 | 5 | connector-consumer-XYZ |
Here I am missing the last 2 lines of log2. I want to modify the query in a way that it produces the table in below manner:
Group | Topic | Partition | Lag | Consumer_ID |
connect-ABC | ABC.sinkevents | 0 | 0 | connector-consumer-ABC |
connect-XYZ | XYZ.cardtransactionauthorizationalertsent | 0 | 5 | connector-consumer-XYZ |
connect-XYZ | XYZ.cardtransactionauthorizationalertsent | 1 | 7 | connector-consumer-XYZ |
connect-XYZ | XYZ.cardtransactionauthorizationalertsent | 2 | 0 | connector-consumer-XYZ |
Please help me to modify the query in a way to get my desired output.
Your kind help on this is highly appreciated.
Thank You..!!
The regex requires each line to begin with "CLIENT-ID\s", which is not the case for the last 2 lines in log2. Removing that requirement in reg101.com matches all lines.
| rex "(?P<Group>[^\s]+)\s(?P<Topic>[^\s]+)\s(?P<Partition>[^\s]+)\s+(?P<Current_Offset>[^\s]+)\s+(?P<Log_End_Offset>[^\s]+)\s+(?P<Lag>[^\s]+)\s+(?P<Consumer_ID>[^\s]+)\s{0,20}(?P<Host>[^\s]+)\s+(?P<Client_ID>[^\s]+)"
Thank you for your suggestion..!! But it is not giving the expected result. May be this is because I missed to update that sometimes there may be other lines before "GROUP" in the logs. My bad..!! Please consider the logs in this way and help me to create the query.
log1:
"connector": {
"state": "RUNNING",
},
"tasks": [
{
"id": 0,
"state": "RUNNING",
}
],
"type": "sink"
}
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
connect-ABC ABC.sinkevents 0 15087148 15087148 0 connector-consumer-ABC /10.231.95.96 connector-consumer-ABC.sinkevents-0
log2:
"connector": {
"state": "RUNNING",
},
"tasks": [
{
"id": 0,
"state": "FAILED",
}
],
"type": "sink"
}
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
connect-XYZ XYZ.cardtransactionauthorizationalertsent 0 27775 27780 5 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 1 27740 27747 7 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 2 27836 27836 0 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
Thank You..!!