Dashboards & Visualizations

How to create a search to pull values from multiple lines?

Mrig342
Contributor

Hi All,

I have logs like below and want to create a table out of it.

 

log1:
    "connector": {
        "state": "RUNNING",
           },
    "tasks": [
        {
            "id": 0,
            "state": "RUNNING",
        }
    ],
    "type": "sink"
}
GROUP                                                                TOPIC                                            PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                                                            HOST            CLIENT-ID
connect-ABC ABC.sinkevents 0          15087148        15087148        0               connector-consumer-ABC /10.231.95.96   connector-consumer-ABC.sinkevents-0

log2:
    "connector": {
        "state": "RUNNING",
           },
    "tasks": [
        {
            "id": 0,
            "state": "FAILED",
        }
    ],
    "type": "sink"
}
GROUP                                                                       TOPIC                                                   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                                                                   HOST            CLIENT-ID
connect-XYZ XYZ.cardtransactionauthorizationalertsent 0          27775           27780           5               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 1          27740           27747           7               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 2          27836           27836           0               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0

 

I created the query which give the below table:

 

.... | rex field=_raw "CLIENT\-ID\s+(?P<Group>[^\s]+)\s(?P<Topic>[^\s]+)\s(?P<Partition>[^\s]+)\s+(?P<Current_Offset>[^\s]+)\s+(?P<Log_End_Offset>[^\s]+)\s+(?P<Lag>[^\s]+)\s+(?P<Consumer_ID>[^\s]+)\s{0,20}(?P<Host>[^\s]+)\s+(?P<Client_ID>[^\s]+)" | table Group,Topic,Partition,Lag,Consumer_ID

 

Group Topic Partition Lag Consumer_ID
connect-ABC ABC.sinkevents 0 0 connector-consumer-ABC
connect-XYZ XYZ.cardtransactionauthorizationalertsent 0 5 connector-consumer-XYZ

Here I am missing the last 2 lines of log2.  I want to modify the query in a way that it produces the table in below manner:

Group Topic Partition Lag Consumer_ID
connect-ABC ABC.sinkevents 0 0 connector-consumer-ABC
connect-XYZ XYZ.cardtransactionauthorizationalertsent 0 5 connector-consumer-XYZ
connect-XYZ XYZ.cardtransactionauthorizationalertsent 1 7 connector-consumer-XYZ
connect-XYZ XYZ.cardtransactionauthorizationalertsent 2 0 connector-consumer-XYZ

Please help me to modify the query in a way to get my desired output.

Your kind help on this is highly appreciated.

Thank You..!!

Labels (1)
Tags (2)
0 Karma
1 Solution

JacekF
Path Finder

The following returns table you are expecting:

|  makeresults
| eval data="    \"connector\": {
        \"state\": \"RUNNING\",
           },
    \"tasks\": [
        {
            \"id\": 0,
            \"state\": \"FAILED\",
        }
    ],
    \"type\": \"sink\"
}
GROUP                                                                       TOPIC                                                   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                                                                   HOST            CLIENT-ID
connect-XYZ XYZ.cardtransactionauthorizationalertsent 0          27775           27780           5               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 1          27740           27747           7               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 2          27836           27836           0               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0"
| rex max_match=0 field=data "\n(?<Group>[^\s]+)\s(?<Topic>[^\s]+)\s(?<Partition>[^\s]+)\s+(?<Current_Offset>[^\s]+)\s+(?<Log_End_Offset>[^\s]+)\s+(?<Lag>[^\s]+)\s+(?<Consumer_ID>[^\s]+)\s*(?<Host>[^\s]+)\s+(?<Client_ID>[^\s]+)"
| table Group,Topic,Partition,Lag,Consumer_ID
| eval Group=mvzip(Group, Topic)
| eval Group=mvzip(Group, Partition)
| eval Group=mvzip(Group, Lag)
| eval Group=mvzip(Group, Consumer_ID)
| fields Group
| mvexpand Group
| makemv Group delim=","
| eval Topic=mvindex(Group, 1)
| eval Partition = mvindex(Group, 2)
| eval Lag = mvindex(Group, 3)
| eval Consumer_ID=mvindex(Group, 4)
| eval Group=mvindex(Group, 0)

 

View solution in original post

Mrig342
Contributor

Hi @JacekF...

Using max_match=0 didn't work.. I tried using max_match=0 after removing "CLIENT-ID\s" and that didn't work either..

Can you please modify it some other way to get the expected result..

Thank you..!!

0 Karma

JacekF
Path Finder

The following returns table you are expecting:

|  makeresults
| eval data="    \"connector\": {
        \"state\": \"RUNNING\",
           },
    \"tasks\": [
        {
            \"id\": 0,
            \"state\": \"FAILED\",
        }
    ],
    \"type\": \"sink\"
}
GROUP                                                                       TOPIC                                                   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                                                                   HOST            CLIENT-ID
connect-XYZ XYZ.cardtransactionauthorizationalertsent 0          27775           27780           5               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 1          27740           27747           7               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 2          27836           27836           0               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0"
| rex max_match=0 field=data "\n(?<Group>[^\s]+)\s(?<Topic>[^\s]+)\s(?<Partition>[^\s]+)\s+(?<Current_Offset>[^\s]+)\s+(?<Log_End_Offset>[^\s]+)\s+(?<Lag>[^\s]+)\s+(?<Consumer_ID>[^\s]+)\s*(?<Host>[^\s]+)\s+(?<Client_ID>[^\s]+)"
| table Group,Topic,Partition,Lag,Consumer_ID
| eval Group=mvzip(Group, Topic)
| eval Group=mvzip(Group, Partition)
| eval Group=mvzip(Group, Lag)
| eval Group=mvzip(Group, Consumer_ID)
| fields Group
| mvexpand Group
| makemv Group delim=","
| eval Topic=mvindex(Group, 1)
| eval Partition = mvindex(Group, 2)
| eval Lag = mvindex(Group, 3)
| eval Consumer_ID=mvindex(Group, 4)
| eval Group=mvindex(Group, 0)

 

Mrig342
Contributor

Hi @JacekF...

Thank you very much for your help on the query..!! This modified query is giving me the expected tabular results.

0 Karma

JacekF
Path Finder

Try add max_match=0 argument to the rex command.

| rex max_match=0 field=_raw <rest of your rex code>

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...