Hi All,
I have logs like below and want to create a table out of it.
log1:
"connector": {
"state": "RUNNING",
},
"tasks": [
{
"id": 0,
"state": "RUNNING",
}
],
"type": "sink"
}
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
connect-ABC ABC.sinkevents 0 15087148 15087148 0 connector-consumer-ABC /10.231.95.96 connector-consumer-ABC.sinkevents-0
log2:
"connector": {
"state": "RUNNING",
},
"tasks": [
{
"id": 0,
"state": "FAILED",
}
],
"type": "sink"
}
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
connect-XYZ XYZ.cardtransactionauthorizationalertsent 0 27775 27780 5 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 1 27740 27747 7 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 2 27836 27836 0 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
I created the query which give the below table:
.... | rex field=_raw "CLIENT\-ID\s+(?P<Group>[^\s]+)\s(?P<Topic>[^\s]+)\s(?P<Partition>[^\s]+)\s+(?P<Current_Offset>[^\s]+)\s+(?P<Log_End_Offset>[^\s]+)\s+(?P<Lag>[^\s]+)\s+(?P<Consumer_ID>[^\s]+)\s{0,20}(?P<Host>[^\s]+)\s+(?P<Client_ID>[^\s]+)" | table Group,Topic,Partition,Lag,Consumer_ID
Group | Topic | Partition | Lag | Consumer_ID |
connect-ABC | ABC.sinkevents | 0 | 0 | connector-consumer-ABC |
connect-XYZ | XYZ.cardtransactionauthorizationalertsent | 0 | 5 | connector-consumer-XYZ |
Here I am missing the last 2 lines of log2. I want to modify the query in a way that it produces the table in below manner:
Group | Topic | Partition | Lag | Consumer_ID |
connect-ABC | ABC.sinkevents | 0 | 0 | connector-consumer-ABC |
connect-XYZ | XYZ.cardtransactionauthorizationalertsent | 0 | 5 | connector-consumer-XYZ |
connect-XYZ | XYZ.cardtransactionauthorizationalertsent | 1 | 7 | connector-consumer-XYZ |
connect-XYZ | XYZ.cardtransactionauthorizationalertsent | 2 | 0 | connector-consumer-XYZ |
Please help me to modify the query in a way to get my desired output.
Your kind help on this is highly appreciated.
Thank You..!!
The following returns table you are expecting:
| makeresults
| eval data=" \"connector\": {
\"state\": \"RUNNING\",
},
\"tasks\": [
{
\"id\": 0,
\"state\": \"FAILED\",
}
],
\"type\": \"sink\"
}
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
connect-XYZ XYZ.cardtransactionauthorizationalertsent 0 27775 27780 5 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 1 27740 27747 7 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 2 27836 27836 0 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0"
| rex max_match=0 field=data "\n(?<Group>[^\s]+)\s(?<Topic>[^\s]+)\s(?<Partition>[^\s]+)\s+(?<Current_Offset>[^\s]+)\s+(?<Log_End_Offset>[^\s]+)\s+(?<Lag>[^\s]+)\s+(?<Consumer_ID>[^\s]+)\s*(?<Host>[^\s]+)\s+(?<Client_ID>[^\s]+)"
| table Group,Topic,Partition,Lag,Consumer_ID
| eval Group=mvzip(Group, Topic)
| eval Group=mvzip(Group, Partition)
| eval Group=mvzip(Group, Lag)
| eval Group=mvzip(Group, Consumer_ID)
| fields Group
| mvexpand Group
| makemv Group delim=","
| eval Topic=mvindex(Group, 1)
| eval Partition = mvindex(Group, 2)
| eval Lag = mvindex(Group, 3)
| eval Consumer_ID=mvindex(Group, 4)
| eval Group=mvindex(Group, 0)
Hi @JacekF...
Using max_match=0 didn't work.. I tried using max_match=0 after removing "CLIENT-ID\s" and that didn't work either..
Can you please modify it some other way to get the expected result..
Thank you..!!
The following returns table you are expecting:
| makeresults
| eval data=" \"connector\": {
\"state\": \"RUNNING\",
},
\"tasks\": [
{
\"id\": 0,
\"state\": \"FAILED\",
}
],
\"type\": \"sink\"
}
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
connect-XYZ XYZ.cardtransactionauthorizationalertsent 0 27775 27780 5 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 1 27740 27747 7 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 2 27836 27836 0 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0"
| rex max_match=0 field=data "\n(?<Group>[^\s]+)\s(?<Topic>[^\s]+)\s(?<Partition>[^\s]+)\s+(?<Current_Offset>[^\s]+)\s+(?<Log_End_Offset>[^\s]+)\s+(?<Lag>[^\s]+)\s+(?<Consumer_ID>[^\s]+)\s*(?<Host>[^\s]+)\s+(?<Client_ID>[^\s]+)"
| table Group,Topic,Partition,Lag,Consumer_ID
| eval Group=mvzip(Group, Topic)
| eval Group=mvzip(Group, Partition)
| eval Group=mvzip(Group, Lag)
| eval Group=mvzip(Group, Consumer_ID)
| fields Group
| mvexpand Group
| makemv Group delim=","
| eval Topic=mvindex(Group, 1)
| eval Partition = mvindex(Group, 2)
| eval Lag = mvindex(Group, 3)
| eval Consumer_ID=mvindex(Group, 4)
| eval Group=mvindex(Group, 0)
Hi @JacekF...
Thank you very much for your help on the query..!! This modified query is giving me the expected tabular results.
Try add max_match=0 argument to the rex command.
| rex max_match=0 field=_raw <rest of your rex code>