Splunk AppDynamics

Dashboards with custom JMX metrics

JohnGregg
Path Finder

All,

The OOTB Kafka JMX configuration is incorrect.  Maybe it was correct for an older version of the Kafka client library, but it's not correct any longer.  For example, there are metrics defined using the object name match pattern "kafka.consumer:type=consumer-fetch-manager-metrics,*" such as records-lag-max.  However records-lag-max is not an attribute of any object matching that pattern.

I have fixed this problem manually and disabled the OOTB configuration.  My new configuration uses the object name match pattern "kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*,topic=*" and the instance identifier "client-id,topic."  My new metrics do show up in the metric browser.  If the client id is myClientId and the topic is myTopic, the metrics show under JMX|ConsumerMetrics|myClientId|myTopic.

However I cannot seem to use the new metrics in a dashboard.  When adding a time series to a dashboard, I can choose "JMX|ProducerMetrics" or "JMX|ConsumerMetrics" as the JMX objects to include.  But when selecting the actual metric to display, I can't see any of my new metrics.  I can only see the OOTB metrics like Average IO Wait.  When selecting the JMX objects to include or when selecting the metrics to display, I cannot drill down further than ProducerMetrics or ConsumerMetrics, even though there are two levels below (corresponding to the client id and topic.)

thanks

Labels (1)
Tags (2)
0 Karma

iamryan
Community Manager
Community Manager

Hi @John.Gregg,

I have not heard back from the Docs team yet. You may want to submit this as a bug in a ticket. I sent you a PM about how your account shows up in community too. 

How do I submit a Support ticket? An FAQ 

0 Karma

iamryan
Community Manager
Community Manager

Hi @John.Gregg,

Thanks for that info. I have shared that with the Docs team. Hopefully, we can also get an answer to your question too.

0 Karma

iamryan
Community Manager
Community Manager

Hi @John.Gregg,

Are you getting this info from an AppD Docs page or something within the product? The more you can share on where this data is found, I can share it with the right team.

0 Karma

JohnGregg
Path Finder

I'm not aware of any detailed documentation on how the OOTB JMX metrics are defined.  There is some high-level documentation but it's wrong, too.  Regardless, I think I've fixed the JMX configuration but I can't select the new metrics for inclusion in a dashboard.

0 Karma
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...