Kafka JMX - Consumer MBean Properties are not fully emitted

[jagadeeshm]

I am using this add-on and configured a new template "KafkaConsumerStats" with the following mbean definition -

<mbean domain="kafka.consumer" properties="type=*,name=*,clientId=*" dumpAllAttributes="true"></mbean>

I enabled JMX on the kafka brokers on port 8081. When i view the jmx emitted data in splunk I only see the following types

kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchRequestRateAndTimeMs,clientId=ReplicaFetcherThread-2-413

kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchResponseSize,clientId=ReplicaFetcherThread-0-413

But none of the other ones that are identified in here under Kafka Consumer Metrics are emitted by JMX.

Kafka Version # 0.8.2.1

I am specifically interested in -
kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=([-.\w]+)

Also, I can't stop noticing that the add-on doesn't have this as a pre-defined template. Is it for the same reason?

Any thoughts?

[jagadeeshm · Accepted solution]

Consumer Metrics are emitted by the consumer application. I originally had the JMX port configured incorrectly and they were not emitted them. This is now resolved.

[lding_splunk]

nice. thanks for the update.

[lding_splunk]

Hi Jagadeeshm,
are you trying the JMX add-on or the dedicated Kafka add-on (on top of the JMX for one channel)? Please try the dedicated add-on for Kafka if not yet.
https://splunkbase.splunk.com/app/2935/

it contains a template for JMX which might be as you need and worth a try (still need you to have JMX installed). Meanwhile, it provides pub-sub channel to get relative data, refer to the doc accordingly.

[jagadeeshm]

This is the one I am using and it doesn't return the above mentioned jmx metrics for kafka consumers.

[lding_splunk]

OK, would you please try the JConsole to double confirm if this kind of data is provided by JMX channel? This will help us to identify if it's an issue with the env or the add-on itself. thanks!

[jagadeeshm]

I will try and confirm on that. But as per the above link I posted, it should be capturing those kafka.consumer properties. On the other hand, is there another way to capture consumer lag information from JMX?

[lding_splunk]

Yeah, it's weird, but definitely something wrong within either the data source (Kafka sides don't provide the data actually) or the add-on sides.

If you have confirmed the data are fetchable via JConsole and probably an issue within the add-on, would you please file a support case to Splunk to provide diag files according for further support?

For your question, the add-on also provides a pub-sub channels on specific Kafka topics but not sure if it can full-fill your requirements.

[jagadeeshm]

I verified jconsole and it doesn't seem to bring up those attributes I mentioned above. Is there a way to get consumer lag information from Kafka JMX metrics?

[jagadeeshm]

https://blog.serverdensity.com/how-to-monitor-kafka/