AppD Archive

Issue: JAVA agent is not reporting to the controller

CommunityUser
Splunk Employee
Splunk Employee

Good day,

We are using a trial version of the APP DYNAMICS which has a lot of features that we are trying to test because we are interested in buying the product. We are preparing a business case for our company so we can buy the software.

We are experiencing issues with the application, specifically with the connection between the agent and the controller, since the JAVA agent (which is installed in a Linux RHL6 and using Jetty as a middleware) is not reporting to the controller installed in a Windows Server 2008. This is a test enviroment and we have modified the configuration xml according to the app dynamics documentation. We are able to see the Machine Agent reporting but the JVM is not reporting to the controller. We've attached a screenshot and also the agent logs so you can look for the issue we are facing.

Thanks in advance for your support, we'll  be waiting for your response.

BR,

Javier Calderón

javier.calderon@avianca.com

0 Karma

Arun_Dasetty
Super Champion

Hi Javier,

We see agent is registered fine but could not find much logging in agent logs as the jvm seems ran for only 1 minuete around 13th Oct 13:53 period, Can you confirm the following to debug further:

- are you starting machine agent with same node name "1stTierNode1" and tier and application details? if possible please send the machine agent logs as well

- is the JVM is running for more than 5 mts as we could not see much logging and hence thought to check the same, if the jvm runtime is short we would need to run with additional jvm arg -Dappdynamics.cron.vm=true added before -javaagent arg

Can you send the output of the following command where appserveragent and machine agent java processing are running:

shell> ps -ef | grep "java"

We might need to start both appserver agent and machine agent with -D jvm arg -Dappdynamics.agent.uniqueHostId=bogcav-tst09 arg and then start machine agent with same arg and need to see how it goes. Can you try this while you send logs requested.

Regards,

Arun

0 Karma

CommunityUser
Splunk Employee
Splunk Employee

Hi, Arun

Thanks for your fast response.  Let me tell you that the JVM seems to be running for more than 5 mts,  We´ve added the additional parameters to the startup command and everything worked fine!  ( -Dappdynamics.cron.vm=true).  We're now running the app agent using the following command line: 

[qauser@bogcav-tst09 javaagent]$ java -Dappdynamics.cron.vm=true -javaagent:/home/javaagent/javaagent.jar=uniqueID=customer1

The problem we have now, is that the agent isn´t stable, it goes down after a couple of minutes and has to be reinitialized manually. 

Also, for some reason we're not able to see the Architecture Map of our App in the Flow Map chart, we have installed App Dynamics for DB (oracle) and properly added in the controller, could you please tell me if there's something we have missed?  

.

Attached you will find the output of the "ps -ef | grep "java" command and the App agent log. 

Thanks for your help!

Br.

0 Karma

CommunityUser
Splunk Employee
Splunk Employee

Hi, Arun
one more thing. Transaction Discovery does not display information, would you please tell me, why can't see business transaction?

image.jpeg

0 Karma

Arun_Dasetty
Super Champion

Hi,

Observations and suggestions:
- We could not locate neither java pid "21554" nor -javaagent:/home/javaagent/javaagent.jar entry and also we do not see machineagent entry process id in psoutput.txt file output
- You are checking at location which is not the screen where discovered BTs are not listed, we request you to check under customer1 -> Business transactions after you
perform the following steps:
a) Navigate to Configure -> Instrumentation -> Transaction Detection -> Click "+" icon under section "Custom Match rules" and define POJO Custom rule
on class and method that you want to monitor as BT and that is invoked for load you made, save the change to rule

Ex:
Class Name (Equals) : amos.server.Main
Method : run

b) Aafter 2-4 minutes generate load on the API or do an agent/jvm restart as since your jvm runs only 5 mts do restart after 2-4 mts after rule
c) Now we expect when you run jvm with -javaagent next time the class/method on which you define rule will be invoked for new load
d) now check in UI whether any BTs discovered under "customer1 -> Business transactions" section in controller UI

Regarding your concern on jvm stability "The problem we have now, is that the agent isn´t stable, it goes down after a couple of minutes and has to be reinitialized manually. " Can you confirm how much long does your JVM run without agent in place , if the jvm runs 10 mts and agent will only show data for 10 mts only.

On a side we see from ps java output you are using another javaagent entry -javaagent:stdclasses/byteman.jar , please be sure we see earlier issues where our agent does not work fine in combination with other -javaagent entry as this could affect Byte code instrumentation of classes in jvm and BT reporting.

We are not seeing any BTs the reason could be you are instrumenting standlone or server which is not ootb supported but we can still define custom Pojo rules and post that you will see flow view and BTs.

Let us know if that information helps

Regards,

Arun

0 Karma

CommunityUser
Splunk Employee
Splunk Employee

Hi, Arun

Thanks for your fast response. Add to log with the information requested. I made the suggested configuration and still not registration transactions:

image.png

image.png

image.png

one more thing. 

reviewing the manual I see that in Preconfigured JMX Metrics, MBean deveria of data capture in this case show no data, us used Jetty in the server and see not this in the jmx metric configuration; you can tell us: is necessary add to configuration or file?

image.png

image.png

Thanks for your help!

0 Karma

Arun_Dasetty
Super Champion

Hi,

We see few disconects here, please be sure that the rule we suggested is an example (refer for only refer for class name amos.server.Main and this class might not exist and you need to change rule to real class name) and not a real example, we request you to update rule to real class existing in JVM instrumentated with -javaagent:/home/javaagent /javaagent.jar entry.

We see the startup script failed to start due to some required args / syntax is not property in java command used in script file start_java_agent.sh, Will it be possible for you to attach the content of the script

In reference to the error the jvm must have exited and the agent will not get chance to instrument the rule hence we do not see any entry point class instrumented in BCT.log

Also we could not correlate the pid with ps java output.

Can you make sure jvm with appdynamics  javagent entry is started fine we see from agent logs that runtime is around 2 minutes due to jvm startup error due to wrong args/syntax and that could be the reason for custom pojo rules not instrumented. Also we see the the option should be class equals and not extending until unless it is run method is implemented in supert class.

Please do the changes accordingly and see how it goes, If you issue please send the sh file and latest debug logs as referred in below screenshots:

image.png

image.png

Regarding your query on JMX, regardless of JMX configuration in Configure -> Instrumentation -> JMX screen for your server we should be able to see mbeans under Node dashboard -> JMX -> Mbean browser if the server exposed mbeans and node is reporting to controller UI. JMX configuration under Configure -> Instrumentatoon screen will only affect jmx metrics in Anayze -> Metric browser, Hope that clarifies your query.

Regards,

Arun

0 Karma

CommunityUser
Splunk Employee
Splunk Employee

Hi AD Team,

EVen I am facing the same issue, how was it resolved?

More information would be helpful.

Regards

Rajendra

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...