Has anyone used Kafka-Connect to Publish Appdynamics Data to Other Source?

[Vaibhav_Vir_Sin]

Experts,

There is an increase demand of APM data to our common data reservoir.

Currently we are extracting data via REST API from Appdynamics, transforming it in the meaningful format and then publishing it to data reservoir. 

But has anyone tried any specific Kafka Connector to stream the AppDynamics data to other sources?

If yes, appreciate if you could share the details over here.

Regards,

Vaibhav

[Subhrajit_mohan]

Hi,

I am new to this. Can you please help me to understand how do you use rest API to extract appdynamics data? If you have any information on this please do share.

Thanks,

Subhrajit

[iamryan]

Hi @Vaibhav Vir.Singh, maybe you can help out @Subhrajit.mohanty  with this question. 

[Vaibhav_Vir_Sin]

HEllo,

Anyone tried above activity by any means?

[David_Hope]

 Hi Vaibhav

I'm very sorry that I haven't got back to you yet, I was hoping to have a little more information.

I will reach out to the internal AppD team to see if we know of anyone who has tried this. I personally unaware of this being tried. Let me do some more investigation.

Happy Holidays.

Kind Regards,

David

[David_Hope]

Hi Vaibhav,

You are currently extracting data via AppDynamics via our REST API. 

Did you want to extract from the REST API->Kafka, or directly out of AppDynamics via Kafka? We don't currently support any other method than via the REST API.

Kind Regards,

David Hope - APM Architect

[Vaibhav_Vir_Sin]

Thanks David for your response.

I am referring Rest API -> Kafka and if anyone has used Kafka connect or
NiFi to integrate.

Currently I am using python script to extract data via rest API,
transforming the data and then publishing to my data lake via AMPS bus and
to be kafka(in simple terms using post API to put the data).

So was thinking if it is feasible and scalable to use any other option to
achieve the same.

[Amit_Jaiswal]

Hi Vaibhav,

I also have a similar requirement to extract AppD data and publish into Data Lake.

Currently, trying to use DEXTER csvs to transform and upload to Data Lake.

But, don't know how to make it Real Time?

Are you able to do it Real Time ? or is there any progress in this since your last query ?

Please guide.

[Vaibhav_Vir_Sin]

No Amit - i am doing in old school way of batch jobs using REST API with 5 mins interval.

Also never heard Appd is working on providing such plugin or functionality where everybody is talking about data streaming.

[Daniel_Odievich]

Vaibhav,

What kind of data are you exfiltrating with your scripts? Metric data? Events? Which ones?

Are you aware of the https://github.com/Appdynamics/AppDynamics.DEXTER/wiki toolkit? It wraps a lot of things you can't ordinarily get into and gives them to you in data warehouse format, ready to go into your data lake.

[Vaibhav_Vir_Sin]

Thanks for your response @Daniel.Odievich

I am subscribing metric data (event data we already getting from AppD-Netcool integration and Netcool data is already onboarded to our data lake).

YEs, I am aware of that but as i have a constraint of using external dependency (from organization security point of view), cannot use that product in Production. Moreover, I felt Dexter is useful for extracting data and potraying the detailed output. But my requirement is somewhat not just extraction but data publishing as well. Where I beleive products\utility like Kafka-Connector for Appdynamics would be of much help and meeting the requirements.

Hope you understand my pain area and looking for someone who already gone through this.

Regards,

vaibhav

[Umervali_Niyama]

I am also extracting data with python scripts. if any one has any thing on this please lwt mw know.

[Umervali_Niyama]

Any thoughts on this?