Need help to create query to pull values from multiple lines

[Mrig342]

Hi All,

I have logs like below and want to create a table out of it.

log1:
GROUP                                                                TOPIC                                            PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                                                            HOST            CLIENT-ID
connect-ABC ABC.sinkevents 0          15087148        15087148        0               connector-consumer-ABC /10.231.95.96   connector-consumer-ABC.sinkevents-0

log2:
GROUP                                                                       TOPIC                                                   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                                                                   HOST            CLIENT-ID
connect-XYZ XYZ.cardtransactionauthorizationalertsent 0          27775           27780           5               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 1          27740           27747           7               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 2          27836           27836           0               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0

I created the query which give the below table:

.... | rex field=_raw "CLIENT\-ID\s+(?P<Group>[^\s]+)\s(?P<Topic>[^\s]+)\s(?P<Partition>[^\s]+)\s+(?P<Current_Offset>[^\s]+)\s+(?P<Log_End_Offset>[^\s]+)\s+(?P<Lag>[^\s]+)\s+(?P<Consumer_ID>[^\s]+)\s{0,20}(?P<Host>[^\s]+)\s+(?P<Client_ID>[^\s]+)" | table Group,Topic,Partition,Lag,Consumer_ID

Group	Topic	Partition	Lag	Consumer_ID
connect-ABC	ABC.sinkevents	0	0	connector-consumer-ABC
connect-XYZ	XYZ.cardtransactionauthorizationalertsent	0	5	connector-consumer-XYZ

Here I am missing the last 2 lines of log2.  I want to modify the query in a way that it produces the table in below manner:

Group	Topic	Partition	Lag	Consumer_ID
connect-ABC	ABC.sinkevents	0	0	connector-consumer-ABC
connect-XYZ	XYZ.cardtransactionauthorizationalertsent	0	5	connector-consumer-XYZ
connect-XYZ	XYZ.cardtransactionauthorizationalertsent	1	7	connector-consumer-XYZ
connect-XYZ	XYZ.cardtransactionauthorizationalertsent	2	0	connector-consumer-XYZ

Please help me to modify the query in a way to get my desired output.

Your kind help on this is highly appreciated.

Thank You..!!

[richgalloway]

The regex requires each line to begin with "CLIENT-ID\s", which is not the case for the last 2 lines in log2.  Removing that requirement in reg101.com matches all lines.

| rex "(?P<Group>[^\s]+)\s(?P<Topic>[^\s]+)\s(?P<Partition>[^\s]+)\s+(?P<Current_Offset>[^\s]+)\s+(?P<Log_End_Offset>[^\s]+)\s+(?P<Lag>[^\s]+)\s+(?P<Consumer_ID>[^\s]+)\s{0,20}(?P<Host>[^\s]+)\s+(?P<Client_ID>[^\s]+)"

[Mrig342]

Hi @richgalloway 

Thank you for your suggestion..!! But it is not giving the expected result. May be this is because I missed to update that sometimes there may be other lines before "GROUP" in the logs. My bad..!! Please consider the logs in this way and help me to create the query.

log1:
    "connector": {
        "state": "RUNNING",
           },
    "tasks": [
        {
            "id": 0,
            "state": "RUNNING",
        }
    ],
    "type": "sink"
}
GROUP                                                                TOPIC                                            PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                                                            HOST            CLIENT-ID
connect-ABC ABC.sinkevents 0          15087148        15087148        0               connector-consumer-ABC /10.231.95.96   connector-consumer-ABC.sinkevents-0

log2:
    "connector": {
        "state": "RUNNING",
           },
    "tasks": [
        {
            "id": 0,
            "state": "FAILED",
        }
    ],
    "type": "sink"
}
GROUP                                                                       TOPIC                                                   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                                                                   HOST            CLIENT-ID
connect-XYZ XYZ.cardtransactionauthorizationalertsent 0          27775           27780           5               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 1          27740           27747           7               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 2          27836           27836           0               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0

Thank You..!!