Dashboards & Visualizations

Need help to create query to pull values from multiple lines

Mrig342
Contributor

Hi All,

I have logs like below and want to create a table out of it.

 

log1:
GROUP                                                                TOPIC                                            PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                                                            HOST            CLIENT-ID
connect-ABC ABC.sinkevents 0          15087148        15087148        0               connector-consumer-ABC /10.231.95.96   connector-consumer-ABC.sinkevents-0

log2:
GROUP                                                                       TOPIC                                                   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                                                                   HOST            CLIENT-ID
connect-XYZ XYZ.cardtransactionauthorizationalertsent 0          27775           27780           5               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 1          27740           27747           7               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 2          27836           27836           0               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0

 

I created the query which give the below table:

 

.... | rex field=_raw "CLIENT\-ID\s+(?P<Group>[^\s]+)\s(?P<Topic>[^\s]+)\s(?P<Partition>[^\s]+)\s+(?P<Current_Offset>[^\s]+)\s+(?P<Log_End_Offset>[^\s]+)\s+(?P<Lag>[^\s]+)\s+(?P<Consumer_ID>[^\s]+)\s{0,20}(?P<Host>[^\s]+)\s+(?P<Client_ID>[^\s]+)" | table Group,Topic,Partition,Lag,Consumer_ID

 

Group Topic Partition Lag Consumer_ID
connect-ABC ABC.sinkevents 0 0 connector-consumer-ABC
connect-XYZ XYZ.cardtransactionauthorizationalertsent 0 5 connector-consumer-XYZ

Here I am missing the last 2 lines of log2.  I want to modify the query in a way that it produces the table in below manner:

Group Topic Partition Lag Consumer_ID
connect-ABC ABC.sinkevents 0 0 connector-consumer-ABC
connect-XYZ XYZ.cardtransactionauthorizationalertsent 0 5 connector-consumer-XYZ
connect-XYZ XYZ.cardtransactionauthorizationalertsent 1 7 connector-consumer-XYZ
connect-XYZ XYZ.cardtransactionauthorizationalertsent 2 0 connector-consumer-XYZ

 

Please help me to modify the query in a way to get my desired output.

Your kind help on this is highly appreciated.

Thank You..!!

Labels (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

The regex requires each line to begin with "CLIENT-ID\s", which is not the case for the last 2 lines in log2.  Removing that requirement in reg101.com matches all lines.

| rex "(?P<Group>[^\s]+)\s(?P<Topic>[^\s]+)\s(?P<Partition>[^\s]+)\s+(?P<Current_Offset>[^\s]+)\s+(?P<Log_End_Offset>[^\s]+)\s+(?P<Lag>[^\s]+)\s+(?P<Consumer_ID>[^\s]+)\s{0,20}(?P<Host>[^\s]+)\s+(?P<Client_ID>[^\s]+)"
---
If this reply helps you, Karma would be appreciated.
0 Karma

Mrig342
Contributor

Hi @richgalloway 

Thank you for your suggestion..!! But it is not giving the expected result. May be this is because I missed to update that sometimes there may be other lines before "GROUP" in the logs. My bad..!! Please consider the logs in this way and help me to create the query.

log1:
    "connector": {
        "state": "RUNNING",
           },
    "tasks": [
        {
            "id": 0,
            "state": "RUNNING",
        }
    ],
    "type": "sink"
}
GROUP                                                                TOPIC                                            PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                                                            HOST            CLIENT-ID
connect-ABC ABC.sinkevents 0          15087148        15087148        0               connector-consumer-ABC /10.231.95.96   connector-consumer-ABC.sinkevents-0

log2:
    "connector": {
        "state": "RUNNING",
           },
    "tasks": [
        {
            "id": 0,
            "state": "FAILED",
        }
    ],
    "type": "sink"
}
GROUP                                                                       TOPIC                                                   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                                                                   HOST            CLIENT-ID
connect-XYZ XYZ.cardtransactionauthorizationalertsent 0          27775           27780           5               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 1          27740           27747           7               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0
connect-XYZ XYZ.cardtransactionauthorizationalertsent 2          27836           27836           0               connector-consumer-XYZ /10.231.95.97   connector-consumer-XYZ.Cardtransactionauthorizationalertsent-0

 

Thank You..!!

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...