How to monitor kafka framework

[CommunityUser]

 I have few servers in clusters for kafka and zookeeper.

What would be the best solution to monitor these two java frameworks.

Whether to install as a standalone java app agent to monitor these to seperately?

Any suggesstions would help me

[Rajesh_Putta]

Hi Sanrajas,

You can go with any of the options listed here

1) Install Java agent to monitor Kafka and Zookeeper. This does Byte Code instrumentation of application libraries and captures performance metrics (code level metrics).

https://docs.appdynamics.com/display/PRO44/Install+the+Java+Agent

Kafka OOTB instrumentation docs

https://docs.appdynamics.com/display/PRO44/Apache+Kafka+Consumer+Backends

Please also refer to Java supported/certified environments and versions

https://docs.appdynamics.com/display/PRO44/Java+Supported+Environments

2) We have extensions available to capture metrics out of those environments and report them. We generally use JMX/REST API to pull metrics out of those environments and report. Please refer following docs link

https://www.appdynamics.com/community/exchange/extension/zookeeper-monitoring-extension/

https://www.appdynamics.com/community/exchange/kafka-monitoring-extension/

Thanks

Rajesh

[CommunityUser]

Hi 

Would you please confirm how we could monitor schema registry & kafka connect as well please. 

Also what's the best practices to integrate AppD with Kafka cluster that monitors producer sending data and consumer consuming data data i.e .wanted to see each of the transaction flowing through end to end in AppD if possible. Is there any Whitepaper you can share i.e. best practices on how to do this. DataDog had a document showing end product as a refernece. I didn't see one for AppDynamics. 

Would be helpful if you could share some steps on 01) creating dashboard that gives overall status for each of the componenent including schema registry/kafka connect etc. 

Cheers

Vijay

[CHANDRA_MOHAN]

Hi,

I do have similar requirement for Monitoring "Topics" in Kafka. Kindly share any update on this requirement

[iamryan]

Hi @CHANDRA.MOHAN,

This is an old post so it may not get a reply from the other members.

Please check out the AppD documentation here: https://docs.appdynamics.com/22.5/search?q=kafka&max=10