How to create a search to pull values from multiple lines?

[Mrig342]

Hi All,

I have logs like below and want to create a table out of it.

log1:
    "connector": {
        "state": "RUNNING",
           },
    "tasks": [
        {
            "id": 0,
            "state": "RUNNING",
        }
    ],
    "type": "sink"
}
GROUP                                                                TOPIC                                            PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                                                            HOST            CLIENT-ID
connect-ABC ABC.sinkevents 0          15087148        15087148        0               connector-consumer-ABC /10.231.95.96   connector-consumer-ABC.sinkevents-0

log2:
    "connector": {
        "state": "RUNNING",
           },
    "tasks": [
        {
            "id": 0,
            "state": "FAILED",
        }
    ],
    "type": "sink"
}
GROUP                                                                       TOPIC                                                   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                                                                   HOST            CLIENT-ID
connect-XYZ XYZ.cardtransactionauthorizationalertsent 0          27775           27780           5               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 1          27740           27747           7               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 2          27836           27836           0               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0

I created the query which give the below table:

.... | rex field=_raw "CLIENT\-ID\s+(?P<Group>[^\s]+)\s(?P<Topic>[^\s]+)\s(?P<Partition>[^\s]+)\s+(?P<Current_Offset>[^\s]+)\s+(?P<Log_End_Offset>[^\s]+)\s+(?P<Lag>[^\s]+)\s+(?P<Consumer_ID>[^\s]+)\s{0,20}(?P<Host>[^\s]+)\s+(?P<Client_ID>[^\s]+)" | table Group,Topic,Partition,Lag,Consumer_ID

Group	Topic	Partition	Lag	Consumer_ID
connect-ABC	ABC.sinkevents	0	0	connector-consumer-ABC
connect-XYZ	XYZ.cardtransactionauthorizationalertsent	0	5	connector-consumer-XYZ

Here I am missing the last 2 lines of log2.  I want to modify the query in a way that it produces the table in below manner:

Group	Topic	Partition	Lag	Consumer_ID
connect-ABC	ABC.sinkevents	0	0	connector-consumer-ABC
connect-XYZ	XYZ.cardtransactionauthorizationalertsent	0	5	connector-consumer-XYZ
connect-XYZ	XYZ.cardtransactionauthorizationalertsent	1	7	connector-consumer-XYZ
connect-XYZ	XYZ.cardtransactionauthorizationalertsent	2	0	connector-consumer-XYZ

Please help me to modify the query in a way to get my desired output.

Your kind help on this is highly appreciated.

Thank You..!!

[Mrig342]

Hi @JacekF...

Using max_match=0 didn't work.. I tried using max_match=0 after removing "CLIENT-ID\s" and that didn't work either..

Can you please modify it some other way to get the expected result..

Thank you..!!

[JacekF · Accepted solution]

The following returns table you are expecting:

|  makeresults
| eval data="    \"connector\": {
        \"state\": \"RUNNING\",
           },
    \"tasks\": [
        {
            \"id\": 0,
            \"state\": \"FAILED\",
        }
    ],
    \"type\": \"sink\"
}
GROUP                                                                       TOPIC                                                   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                                                                   HOST            CLIENT-ID
connect-XYZ XYZ.cardtransactionauthorizationalertsent 0          27775           27780           5               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 1          27740           27747           7               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 2          27836           27836           0               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0"
| rex max_match=0 field=data "\n(?<Group>[^\s]+)\s(?<Topic>[^\s]+)\s(?<Partition>[^\s]+)\s+(?<Current_Offset>[^\s]+)\s+(?<Log_End_Offset>[^\s]+)\s+(?<Lag>[^\s]+)\s+(?<Consumer_ID>[^\s]+)\s*(?<Host>[^\s]+)\s+(?<Client_ID>[^\s]+)"
| table Group,Topic,Partition,Lag,Consumer_ID
| eval Group=mvzip(Group, Topic)
| eval Group=mvzip(Group, Partition)
| eval Group=mvzip(Group, Lag)
| eval Group=mvzip(Group, Consumer_ID)
| fields Group
| mvexpand Group
| makemv Group delim=","
| eval Topic=mvindex(Group, 1)
| eval Partition = mvindex(Group, 2)
| eval Lag = mvindex(Group, 3)
| eval Consumer_ID=mvindex(Group, 4)
| eval Group=mvindex(Group, 0)

[Mrig342]

Hi @JacekF...

Thank you very much for your help on the query..!! This modified query is giving me the expected tabular results.

[JacekF]

Try add max_match=0 argument to the rex command.

| rex max_match=0 field=_raw <rest of your rex code>