All, The OOTB Kafka JMX configuration is incorrect. Maybe it was correct for an older version of the Kafka client library, but it's not correct any longer. For example, there are metrics defined using the object name match pattern "kafka.consumer:type=consumer-fetch-manager-metrics,*" such as records-lag-max. However records-lag-max is not an attribute of any object matching that pattern. I have fixed this problem manually and disabled the OOTB configuration. My new configuration uses the object name match pattern "kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*,topic=*" and the instance identifier "client-id,topic." My new metrics do show up in the metric browser. If the client id is myClientId and the topic is myTopic, the metrics show under JMX|ConsumerMetrics|myClientId|myTopic. However I cannot seem to use the new metrics in a dashboard. When adding a time series to a dashboard, I can choose "JMX|ProducerMetrics" or "JMX|ConsumerMetrics" as the JMX objects to include. But when selecting the actual metric to display, I can't see any of my new metrics. I can only see the OOTB metrics like Average IO Wait. When selecting the JMX objects to include or when selecting the metrics to display, I cannot drill down further than ProducerMetrics or ConsumerMetrics, even though there are two levels below (corresponding to the client id and topic.) thanks
... View more