All Apps and Add-ons

Kafka JMX - Consumer MBean Properties are not fully emitted

jagadeeshm
Contributor

I am using this add-on and configured a new template "KafkaConsumerStats" with the following mbean definition -

<mbean domain="kafka.consumer" properties="type=*,name=*,clientId=*" dumpAllAttributes="true"></mbean>

I enabled JMX on the kafka brokers on port 8081. When i view the jmx emitted data in splunk I only see the following types

kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchRequestRateAndTimeMs,clientId=ReplicaFetcherThread-2-413

kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchResponseSize,clientId=ReplicaFetcherThread-0-413

But none of the other ones that are identified in here under Kafka Consumer Metrics are emitted by JMX.

Kafka Version # 0.8.2.1

I am specifically interested in -
kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=([-.\w]+)

Also, I can't stop noticing that the add-on doesn't have this as a pre-defined template. Is it for the same reason?

Any thoughts?

0 Karma
1 Solution

jagadeeshm
Contributor

Consumer Metrics are emitted by the consumer application. I originally had the JMX port configured incorrectly and they were not emitted them. This is now resolved.

View solution in original post

0 Karma

jagadeeshm
Contributor

Consumer Metrics are emitted by the consumer application. I originally had the JMX port configured incorrectly and they were not emitted them. This is now resolved.

0 Karma

lding_splunk
Splunk Employee
Splunk Employee

nice. thanks for the update.

0 Karma

lding_splunk
Splunk Employee
Splunk Employee

Hi Jagadeeshm,
are you trying the JMX add-on or the dedicated Kafka add-on (on top of the JMX for one channel)? Please try the dedicated add-on for Kafka if not yet.
https://splunkbase.splunk.com/app/2935/

it contains a template for JMX which might be as you need and worth a try (still need you to have JMX installed). Meanwhile, it provides pub-sub channel to get relative data, refer to the doc accordingly.

0 Karma

jagadeeshm
Contributor

This is the one I am using and it doesn't return the above mentioned jmx metrics for kafka consumers.

0 Karma

lding_splunk
Splunk Employee
Splunk Employee

OK, would you please try the JConsole to double confirm if this kind of data is provided by JMX channel? This will help us to identify if it's an issue with the env or the add-on itself. thanks!

0 Karma

jagadeeshm
Contributor

I will try and confirm on that. But as per the above link I posted, it should be capturing those kafka.consumer properties. On the other hand, is there another way to capture consumer lag information from JMX?

0 Karma

lding_splunk
Splunk Employee
Splunk Employee

Yeah, it's weird, but definitely something wrong within either the data source (Kafka sides don't provide the data actually) or the add-on sides.

If you have confirmed the data are fetchable via JConsole and probably an issue within the add-on, would you please file a support case to Splunk to provide diag files according for further support?

For your question, the add-on also provides a pub-sub channels on specific Kafka topics but not sure if it can full-fill your requirements.

0 Karma

jagadeeshm
Contributor

I verified jconsole and it doesn't seem to bring up those attributes I mentioned above. Is there a way to get consumer lag information from Kafka JMX metrics?

0 Karma

jagadeeshm
Contributor
0 Karma
Get Updates on the Splunk Community!

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...