topic Re: Need help to crate query to pull values from multiple lines in Dashboards & Visualizations

How to create a search to pull values from multiple lines?

Mrig342 — Mon, 25 Jul 2022 16:05:57 GMT

Hi All,

I have logs like below and want to create a table out of it.

log1: "connector": { "state": "RUNNING", }, "tasks": [ { "id": 0, "state": "RUNNING", } ], "type": "sink" } GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID connect-ABC ABC.sinkevents 0 15087148 15087148 0 connector-consumer-ABC /10.231.95.96 connector-consumer-ABC.sinkevents-0 log2: "connector": { "state": "RUNNING", }, "tasks": [ { "id": 0, "state": "FAILED", } ], "type": "sink" } GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID connect-XYZ XYZ.cardtransactionauthorizationalertsent 0 27775 27780 5 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0 connect-XYZ XYZ.cardtransactionauthorizationalertsent 1 27740 27747 7 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0 connect-XYZ XYZ.cardtransactionauthorizationalertsent 2 27836 27836 0 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0

I created the query which give the below table:

.... | rex field=_raw "CLIENT\-ID\s+(?P<Group>[^\s]+)\s(?P<Topic>[^\s]+)\s(?P<Partition>[^\s]+)\s+(?P<Current_Offset>[^\s]+)\s+(?P<Log_End_Offset>[^\s]+)\s+(?P<Lag>[^\s]+)\s+(?P<Consumer_ID>[^\s]+)\s{0,20}(?P<Host>[^\s]+)\s+(?P<Client_ID>[^\s]+)" | table Group,Topic,Partition,Lag,Consumer_ID

Group	Topic	Partition	Lag	Consumer_ID
connect-ABC	ABC.sinkevents	0	0	connector-consumer-ABC
connect-XYZ	XYZ.cardtransactionauthorizationalertsent	0	5	connector-consumer-XYZ

Here I am missing the last 2 lines of log2. I want to modify the query in a way that it produces the table in below manner:

Group	Topic	Partition	Lag	Consumer_ID
connect-ABC	ABC.sinkevents	0	0	connector-consumer-ABC
connect-XYZ	XYZ.cardtransactionauthorizationalertsent	0	5	connector-consumer-XYZ
connect-XYZ	XYZ.cardtransactionauthorizationalertsent	1	7	connector-consumer-XYZ
connect-XYZ	XYZ.cardtransactionauthorizationalertsent	2	0	connector-consumer-XYZ

Please help me to modify the query in a way to get my desired output.

Your kind help on this is highly appreciated.

Thank You..!!

Re: Need help to crate query to pull values from multiple lines

JacekF — Mon, 25 Jul 2022 11:01:55 GMT

Try add max_match=0 argument to the rex command.

| rex max_match=0 field=_raw <rest of your rex code>

Re: Need help to crate query to pull values from multiple lines

Mrig342 — Mon, 25 Jul 2022 12:43:49 GMT

Hi @JacekF...

Using max_match=0 didn't work.. I tried using max_match=0 after removing "CLIENT-ID\s" and that didn't work either..

Can you please modify it some other way to get the expected result..

Thank you..!!

Re: Need help to crate query to pull values from multiple lines

JacekF — Mon, 25 Jul 2022 16:03:38 GMT

The following returns table you are expecting:

| makeresults | eval data=" \"connector\": { \"state\": \"RUNNING\", }, \"tasks\": [ { \"id\": 0, \"state\": \"FAILED\", } ], \"type\": \"sink\" } GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID connect-XYZ XYZ.cardtransactionauthorizationalertsent 0 27775 27780 5 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0 connect-XYZ XYZ.cardtransactionauthorizationalertsent 1 27740 27747 7 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0 connect-XYZ XYZ.cardtransactionauthorizationalertsent 2 27836 27836 0 connector-consumer-XYZ /10.231.95.97 connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0" | rex max_match=0 field=data "\n(?<Group>[^\s]+)\s(?<Topic>[^\s]+)\s(?<Partition>[^\s]+)\s+(?<Current_Offset>[^\s]+)\s+(?<Log_End_Offset>[^\s]+)\s+(?<Lag>[^\s]+)\s+(?<Consumer_ID>[^\s]+)\s*(?<Host>[^\s]+)\s+(?<Client_ID>[^\s]+)" | table Group,Topic,Partition,Lag,Consumer_ID | eval Group=mvzip(Group, Topic) | eval Group=mvzip(Group, Partition) | eval Group=mvzip(Group, Lag) | eval Group=mvzip(Group, Consumer_ID) | fields Group | mvexpand Group | makemv Group delim="," | eval Topic=mvindex(Group, 1) | eval Partition = mvindex(Group, 2) | eval Lag = mvindex(Group, 3) | eval Consumer_ID=mvindex(Group, 4) | eval Group=mvindex(Group, 0)

Re: Need help to crate query to pull values from multiple lines

Mrig342 — Tue, 26 Jul 2022 08:38:08 GMT

Hi @JacekF...

Thank you very much for your help on the query..!! This modified query is giving me the expected tabular results.