topic Need help to create query to pull values from multiple lines in Dashboards & Visualizations

Need help to create query to pull values from multiple lines

Mrig342 — Mon, 25 Jul 2022 14:56:53 GMT

Hi All,

I have logs like below and want to create a table out of it.

log1: GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID connect-ABC ABC.sinkevents 0 15087148 15087148 0 connector-consumer-ABC /10.231.95.96 connector-consumer-ABC.sinkevents-0 log2: GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID connect-XYZ XYZ.cardtransactionauthorizationalertsent 0 27775 27780 5 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0 connect-XYZ XYZ.cardtransactionauthorizationalertsent 1 27740 27747 7 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0 connect-XYZ XYZ.cardtransactionauthorizationalertsent 2 27836 27836 0 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0

I created the query which give the below table:

.... | rex field=_raw "CLIENT\-ID\s+(?P<Group>[^\s]+)\s(?P<Topic>[^\s]+)\s(?P<Partition>[^\s]+)\s+(?P<Current_Offset>[^\s]+)\s+(?P<Log_End_Offset>[^\s]+)\s+(?P<Lag>[^\s]+)\s+(?P<Consumer_ID>[^\s]+)\s{0,20}(?P<Host>[^\s]+)\s+(?P<Client_ID>[^\s]+)" | table Group,Topic,Partition,Lag,Consumer_ID

Group	Topic	Partition	Lag	Consumer_ID
connect-ABC	ABC.sinkevents	0	0	connector-consumer-ABC
connect-XYZ	XYZ.cardtransactionauthorizationalertsent	0	5	connector-consumer-XYZ

Here I am missing the last 2 lines of log2. I want to modify the query in a way that it produces the table in below manner:

Group	Topic	Partition	Lag	Consumer_ID
connect-ABC	ABC.sinkevents	0	0	connector-consumer-ABC
connect-XYZ	XYZ.cardtransactionauthorizationalertsent	0	5	connector-consumer-XYZ
connect-XYZ	XYZ.cardtransactionauthorizationalertsent	1	7	connector-consumer-XYZ
connect-XYZ	XYZ.cardtransactionauthorizationalertsent	2	0	connector-consumer-XYZ

Please help me to modify the query in a way to get my desired output.

Your kind help on this is highly appreciated.

Thank You..!!

Re: Need help to crate query to pull values from multiple lines

richgalloway — Sun, 24 Jul 2022 13:49:17 GMT

The regex requires each line to begin with "CLIENT-ID\s", which is not the case for the last 2 lines in log2. Removing that requirement in reg101.com matches all lines.

| rex "(?P<Group>[^\s]+)\s(?P<Topic>[^\s]+)\s(?P<Partition>[^\s]+)\s+(?P<Current_Offset>[^\s]+)\s+(?P<Log_End_Offset>[^\s]+)\s+(?P<Lag>[^\s]+)\s+(?P<Consumer_ID>[^\s]+)\s{0,20}(?P<Host>[^\s]+)\s+(?P<Client_ID>[^\s]+)"

Re: Need help to crate query to pull values from multiple lines

Mrig342 — Sun, 24 Jul 2022 17:24:11 GMT

Hi @richgalloway

Thank you for your suggestion..!! But it is not giving the expected result. May be this is because I missed to update that sometimes there may be other lines before "GROUP" in the logs. My bad..!! Please consider the logs in this way and help me to create the query.

log1: "connector": { "state": "RUNNING", }, "tasks": [ { "id": 0, "state": "RUNNING", } ], "type": "sink" } GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID connect-ABC ABC.sinkevents 0 15087148 15087148 0 connector-consumer-ABC /10.231.95.96 connector-consumer-ABC.sinkevents-0 log2: "connector": { "state": "RUNNING", }, "tasks": [ { "id": 0, "state": "FAILED", } ], "type": "sink" } GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID connect-XYZ XYZ.cardtransactionauthorizationalertsent 0 27775 27780 5 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0 connect-XYZ XYZ.cardtransactionauthorizationalertsent 1 27740 27747 7 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0 connect-XYZ XYZ.cardtransactionauthorizationalertsent 2 27836 27836 0 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0

Thank You..!!