Machine agent and App server agent are not working Simultaneously for Cassandra DB Nodes

[CommunityUser]

Hi,

I'm facing a strange issue on appdynamics machineagent for Cassandra DB nodes of a java application. Can somebody help to overcome this issue? I've followed the same process for tomcat nodes and installation was successful. I'm explaining the process below I've followed.

I've App Server agents are up and running, reporting fine to appdynamics for Cassandra DB nodes. Then I copied all the required files to install Machine Agent (eg. controller-info.xml, machineagent.jar etc.) maintaining proper folder structure in Cassandra DB server. As App server agents are already running and nodes are already configured I don't need to specify the Node details in controller-info of the machine agent. Now I run machine agent with command:java -Xms64m -Xmx64m -jar machineagent.jar &. 

Then I saw in Appdynamics that Machine agent is created but not assigned to the application(It should not happen as I already have Application,Tier,Nodes and App Server Agents running). However I assigned the Application, Tier, node manually to the Machine Agent. 

Then I faced the stange issue that when Machine Agent gets up and starts reporting, App server agent is going down and stops reporting. If I restart the Cassandra DSE service the App server agent goes up and starts reporting. In other hand the association(with Application, Tier, node) of the Machine Agent has been removed. 

I've taken the log when App Server agent goes down and finally stops reporting(PFA). PFB some extract from the log file:

Error in controller in processing binary request BT Registration Data - Application component node id:126812 is not associated to machine id:53603
[AD Thread-Metric Reporter0] 17 Nov 2014 06:47:03,311 WARN MetricHandler - Metric Reporter Queue full. Dropping metrics.

Thanks.

[CommunityUser]

Attaching AppServer Agent Logs...

[Arun_Dasetty]

Hi,

1) We see from agent logs that you are trying to start same -javaagent:/etc/dse/appdynamicsDemo/javaagent.jar entry from two different JVM instances at different times and that should be fine but the problem you are using same app/tier and node-name for both the JVM instance and that will affect other agent node registeration and hence we see in AppAgentLog1.txt file the ResponseReadException exceptin as other agent started registered and this is reuslting as duplicate node:

[Thread-0] 26 Nov 2014 06:48:45,883 INFO ConfigurationChannel - Auto agent registration attempted: Application Name [GEDS-Stg] Component Name [Cassandra DB] Node Name [Cassandra DB Node 4]
...[AD Thread Pool-Global15] 26 Nov 2014 13:31:26,690 INFO BusinessTransactionRegistry - Sending transactions to register [BusinessTransaction{id=0,
[Thread-0] 26 Nov 2014 13:31:17,604 INFO ConfigurationChannel - Registration information received Node ID[126407] Component ID[4857] Application ID [1133]

from AppAgentLog2.txt:
[Thread-0] 26 Nov 2014 13:31:13,417 INFO AgentKernel - Starting Java Agent at Wed Nov 26 13:31:13 UTC 2014 ...
[Thread-0] 26 Nov 2014 13:31:13,538 INFO XMLConfigManager - Default Host Identifier Resolver using host name for unique host identifier [ip-10-37-11-107.us-west-1.compute.internal]
[Thread-0] 26 Nov 2014 13:31:17,603 INFO ConfigurationChannel - Auto agent registration attempted: Application Name [GEDS-Stg] Component Name [Cassandra DB] Node Name [Cassandra DB Node 4]

2) We can see logging and multiple JVM PID in agent logs provided and we can see logging in AppAgentLog2.txt file with same app/tier and node name and same uniquehost name starting at 26 Nov 2014 13:31:13,417 where during same time in AppAgentLog1.txt agent got registration error as the other jvm using same node name

3) Regarding the machine agent logs we do not locate logs are useful as they do not contain node name and host or jvm args logging, request you to provide logs collected from machine agent after machine agent java process restart if the following suggestion does not help

4) Plan of action:

- please stop the appserver agent and machine agent that having host name 10.37.11.73 as part host name value

- start the appserver with -Dappdynamics.agent.uniqueHostId=ip-10.37.11.73.us.west-1.com and see it is reporting fine in UI for corresponding agent node 

- now start machine agent with same app/tier and node details as in appagent node in above step for which we want to associate this machine agent node from ip 10.37.11.73 as follows:

shell> java -Dappdynamics.agent.uniqueHostId=ip-10.37.11.73.us.west-1.com -jar machineagent.jar

now see how it goes after above step

- If you want to associate machine agent node to particular node in app [GEDS-Stg] tier [Cassandra DB] Node Name [Cassandra DB Node 4] than please do following:

- stop appserver agent node reporting to [Cassandra DB Node 4] (make sure on jvm uses unique node-name) and start with additional -D jvm arg in addition to -javaagent entry:

-Dappdynamics.agent.uniqueHostId=ip-10-37-11-107.us-west-1.compute.internal

- now start machine agent you want to associate to node [Cassandra DB Node 4] under same app/tier/node in above step as follows:

shell> java -Dappdynamics.agent.uniqueHostId=ip-10-37-11-107.us-west-1.compute.internal -jar machineagent.jar

If you want another machine agent node to reporting to [Cassandra DB Node 4] repeat above steps but make sure uniqueHostId for each appserver agent is unique across other agent nodes and machine agent should use same uniquehost that is used to associate appserver agnet node for which you want to associate machine agent node

Let us know if that information helps.

Refer doc for steps to associate machine agent from different host to particular agent node in UI:

https://docs.appdynamics.com/display/PRO39/Configure+Multiple+Standalone+Machine+Agents+for+One+Mach...

 Regards,

Arun

[Arun_Dasetty]

Hi,

We do agree that you do not need to specify node/tier details in machine agent controller-info.xml file but this is when in the following cases:

- no multiple appserver agent nodes are associated to controller ui under different apps

- appserver agent node and machine agent are resolved to different unique host name somehow

- machine agent is trying same node name like appserer agent to register but with same -Dappdynamics.agent.uniqueHostId=<uniqueHostname> jvm arg used by appserver agent . in this case restarting with same uniqueHostId arg should fix

To see whether above is the case, please send the following to assist you further:

- screenshot from controller UI with application-name as search filter under both agents and machine agents section in screen Setup -> AppDynamics Agents

- latest agent.log under <appserveragent_dir>/logs path and machine agent logs  under <machineagent_dir>/logs path to see jvm args and agent config details used to register and host resolved in both agents

refer doc for more information : https://docs.appdynamics.com/display/PRO39/Configure+Multiple+Standalone+Machine+Agents+for+One+Mach...

Keep us posted requested information to debug further. 

Regards,

Arun

[CommunityUser]

Thanks Arun.

Attaching AppServer Agent and Machine Agent Logs and UI screenshots. I'm giving some older machine agent log but it is the same as current one. In App server Agent log I'm concened about one line which I mentioned earliar as well.

"There is no component associated with node id:126407 and machine id:53295
[AD Thread-Metric Reporter0] 26 Nov 2014 12:31:22,310 WARN MetricHandler - Metric Reporter Queue full. Dropping metrics."

Let me know if you need more details.

[Arun_Dasetty]

Hi,

Please attach the logs and screenshots from Setup -> appdynamics agents screen as requested earlier to assist you further, we do not locate logs in your post

Regards,

Arun