topic Re: Jboss 7 monitoring only shows Memory data. in AppD Archive

Jboss 7 monitoring only shows Memory data.

CommunityUser — Tue, 06 Aug 2013 12:28:54 GMT

I have followed the manual steps to configure monitoring on a JBOSS 7.1.1, after all steps, my controller only shows Memory and garbage collector metrics for the monitored JBOSS instance.

Have you ever seen this before? What can it be?

Re: Jboss 7 monitoring only shows Memory data.

CommunityUser — Wed, 07 Aug 2013 19:56:50 GMT

Below are the java versions if it helps to find what is happening.

java version "1.7.0_25"
Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)

Re: Jboss 7 monitoring only shows Memory data.

Arun_Dasetty — Thu, 08 Aug 2013 08:52:21 GMT

Hi,

Thanks for writing in. We confirm we have customer monitoring JBoss 7 with successfully monitoring business transactions under BT with call stats fine under node dashboard -> dashbpard as opposed to screenshot "app1.JPG"

Can you please provide the following details:
- we see such issues if agent is disabled under node dashboard -> agents -> "appserver agent" screen, we assume that agent is enabled under "appserver agent" section in reference to screenshot "app3.JPG" in this case, let us know if any disconnect.
- Can you confirm whether you have generated load on the jboss server (did you make any business transaction or requests like servlets, jms etc ) on the server with agent
- Can you please send the zipped version logs folder under <agent_install_dir>/logs, we would like to check if any errors realted to BT detection

Please keep us posted the requested details.

Thanks,

Arun

Re: Jboss 7 monitoring only shows Memory data.

CommunityUser — Thu, 08 Aug 2013 11:32:49 GMT

Requested logs follows attached.

Thanks a lot

Re: Jboss 7 monitoring only shows Memory data.

Arun_Dasetty — Thu, 08 Aug 2013 11:38:46 GMT

Hi ,

We see that you have sent controller server.log, we request you to send logs under <agent_install_dir>/logs, also you can select "all logs in the agent directory" option (the first option in attached screenshot) from jboss node directory

we request you to revert with logs under agent directory, where agent dir is the path you have confiugred for -javaagent: parameter in jboss startup script.

Thanks,

Arun

Re: Jboss 7 monitoring only shows Memory data.

CommunityUser — Thu, 08 Aug 2013 11:44:30 GMT

Ok, our server is down right now, I´ll send you the correct logs as soon as it is available to me.

Thank you.

Re: Jboss 7 monitoring only shows Memory data.

Arun_Dasetty — Thu, 08 Aug 2013 11:45:31 GMT

Hi,

Thanks for the update. sure keep us posted the requested logs.

Thanks,

Arun

Re: Jboss 7 monitoring only shows Memory data.

CommunityUser — Thu, 08 Aug 2013 15:35:39 GMT

Follows attached the agent logs.

thanks.

Re: Jboss 7 monitoring only shows Memory data.

Arun_Dasetty — Thu, 08 Aug 2013 18:20:51 GMT

Hi,

Thanks for posting back, we see different errors in agent.log, and we see the connection refused error repetively in all agent.logs:
----------------------
[AD Thread Pool-Global1] 08 Aug 2013 09:38:46,225 ERROR ConfigurationChannel - Fatal transport error: Connection refused
[AD Thread Pool-Global1] 08 Aug 2013 09:38:46,225 WARN ConfigurationChannel - Invalid response for configuration request from controller/could not connect to controller. Msg : Fatal transport error: Connection refused
..
[AD Thread Pool-Global0] 08 Aug 2013 09:39:07,415 ERROR ConfigurationChannel - Fatal transport error: Connection refused
[AD Thread Pool-Global0] 08 Aug 2013 09:39:07,415 WARN ConfigurationChannel - Could not connect to the controller/invalid response from controller, cannot get initialization information, controller host [127.0.0.1], port[8090], exception [Fatal transport error: Connection refused]
-----------------------

we would like to inform you that, It is not recommended to provide controller host name as 127.0.0.1,

- Can you confirm whether controller is running in the same host wheher jboss server with agent is running

Can you confirm whether controller is accessible currently?

- Is your jboss server is running domain mode or standalone mode and what is the jboss version?

- Can you confirm whether you have configured jboss seever as per docs, if the above said is right can you provide the script/command you are using to start jboss server?
http://docs.appdynamics.com/display/PRO13S/JBoss+Startup+Settings#JBossStartupSettings-ToaddthejavaagentcommandinaLinuxenvironmentforJBossAS7.x

We requested the above information as we see two different agent logs with different jvm process id from which we suspect that agent is starting twice or not configured on domain with application deployed.
-----------------------------------------------------------------
from agent.2013_08_08__09_17_04.0.log:
[Thread-0] 08 Aug 2013 08:45:23,085 INFO ConfigurationChannel - JVM PID: 6127
[AD Thread-Metric Reporter0] 08 Aug 2013 09:39:02,055 WARN MetricHandler - Metric Reporter Queue full. Dropping metrics.

from agent.2013_08_08__08_45_21.0.log
[Thread-0] 08 Aug 2013 08:45:21,403 INFO AgentKernel - Starting Java Agent at Thu Aug 08 08:45:21 BRT 2013 ...
[Thread-0] 08 Aug 2013 09:17:07,036 INFO ConfigurationChannel - JVM PID: 6492
[AD Thread Pool-Global1] 08 Aug 2013 09:38:23,598 ERROR ConfigurationChannel - Fatal transport error: Connection refused

----------------------------------------------------------------

Thanks,

Arun

Re: Jboss 7 monitoring only shows Memory data.

CommunityUser — Thu, 08 Aug 2013 18:40:44 GMT

Hi Akumar.

The monitored server is connected to the controler via reverse SSH tunnel, those connection errors ocurred because the reverse tunnel was not enabled, when I enable it, the message stops. here is the reverse ssh tunnel command (ssh -vvv -nNT -R 8090:127.0.0.1:8090 blkhmlbrb@172.23.2.142) thats why the server is using 127.0.0.1 as controller IP.

Jboss is version 7 and running on standalone mode.

standalone.sh follows attached.

Re: Jboss 7 monitoring only shows Memory data.

Arun_Dasetty — Sat, 10 Aug 2013 07:34:59 GMT

Hi,

Thanks for posting back, we see that you are using jboss-as-7.1.1 version, and can you you please confirm whether you have configure standalone.conf and have added the following entry specific to jboss 7.1.1 version as per docs:

For JBoss 7.1.1, add the additional packages as shown here:

JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,com.appdynamics,com.appdynamics.,com.singularity,com.singularity.,org.jboss.logmanager"
 
-
http://docs.appdynamics.com/display/PRO13S/JBoss+Startup+Settings#JBossStartupSettings-ToaddthejavaagentcommandinaLinuxenvironmentforJBossAS7.x

and also make sure that you have configured as per steps 1,2,3 as per docs and if you still do not see any BTs discovered after making load on jboss server, can you confirm whether you the URL is hitting is servlet request, as the static html requests will not be monitored by agent, if the issue still persists send us the latest agent logs and standalone.conf

Thanks,

Arun

Re: Jboss 7 monitoring only shows Memory data.

CommunityUser — Mon, 12 Aug 2013 11:20:56 GMT

Hello!

Thanks for your reply.

I did all the steps you mentioned on your message, standalone.conf is attached.

Yes there are transactions going on the server, including lots of oracle database queries as I could see on the APP Server Log.

Re: Jboss 7 monitoring only shows Memory data.

CommunityUser — Mon, 12 Aug 2013 11:29:26 GMT

Here is the java app running.

[blkhmlbrb@hib340210 share]$ ps -ef |grep java
514 3734 3676 0 Aug09 ? 00:29:00 /usr/java/jdk1.7.0_25/bin/java -D[Standalone] -XX:+UseCompressedOops -XX:+TieredCompilation -server -Dprog.name=BRBBanklinkJboss -Xms2048m -Xmx2048m -XX:MaxPermSize=512M -Djboss.server.log.dir=/home/blkhmlbrb/jboss-deploy/logs/ -Dfile.encoding=ISO-8859-1 -Dlog4j.debug=true -Djava.awt.headless=true -XX:+UseParallelGC -XX:SurvivorRatio=6 -Djboss.modules.system.pkgs=org.jboss.byteman,com.appdynamics,com.appdynamics.,com.singularity,com.singularity.,org.jboss.logmanager -Djava.awt.headless=true -XX:-UseSplitVerifier -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:/usr/share/jboss-as/jboss-as-7.1.1.Final/modules/org/jboss/logmanager/main/jboss-logmanager-1.2.2.GA.jar:/usr/share/jboss-as/jboss-as-7.1.1.Final/modules/org/jboss/logmanager/log4j/main/jboss-logmanager-log4j-1.0.0.GA.jar:/usr/share/jboss-as/jboss-as-7.1.1.Final/modules/org/apache/log4j/main/log4j-1.2.16.jar -javaagent:/home/blkhmlbrb/apps/AppServerAgent/javaagent.jar -Dorg.jboss.boot.log.file=/var/foton/blkhmlbrb/logs/boot.log -Dlogging.configuration=file:/usr/share/jboss-as/jboss-as-7.1.1.Final/standalone/configuration/logging.properties -jar /usr/share/jboss-as/jboss-as-7.1.1.Final/jboss-modules.jar -mp /usr/share/jboss-as/jboss-as-7.1.1.Final/modules -jaxpmodule javax.xml.jaxp-provider -logmodule org.jboss.logmanager.LogManager org.jboss.as.standalone -Djboss.home.dir=/usr/share/jboss-as/jboss-as-7.1.1.Final -b 0.0.0.0

Re: Jboss 7 monitoring only shows Memory data.

CommunityUser — Mon, 12 Aug 2013 12:11:22 GMT

Agent Logs attached.

only those 2 files had changes today.

All the others are still modified dated as friday

Re: Jboss 7 monitoring only shows Memory data.

Arun_Dasetty — Mon, 12 Aug 2013 12:48:50 GMT

Hi,

Thanks for posting back and providing logs, Can you please try as follows:

- we still see the errors which must be resolved as this affect the reporting of agent metrics, we understood that you have mentioned that you have set controler via reverse SSH tunnel, but we are expecting that the following errors should be resolved and should not occure recursively in agent.log:

-----------------
[AD Thread-Metric Reporter0] 12 Aug 2013 08:48:48,325 ERROR MetricHandler - Error sending metrics - will requeue for later transmission
com.singularity.ee.agent.commonservices.metricgeneration.metrics.g: Fatal transport error: Connection refused
-----------

- Can you remove the following option from jvm_opts "-Dorg.jboss.boot.log.file=/var/foton/blkhmlbrb/logs/boot.log " and restart jboss

- We see the following entry points applied successfully from BCI.log:
=================================
    Line 5858:        Applying method interceptor entry.Servlet at org/jboss/resteasy/plugins/server/servlet/HttpServletDispatcher.service ((Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse; ) V) id:3
   Line 8321:        Applying method interceptor entry.webservice.jboss.ws.impl.Service at org/jboss/ws/core/server/ServiceEndpointInvoker.invoke ((Lorg/jboss/wsf/spi/invocation/InvocationContext; ) V ) id:7
   Line 9381:        Applying method interceptor entry.Servlet at org/jboss/as/web/WelcomeContextConsoleServlet.service ((Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse; ) V) id:8
   Line 12869:        Applying method interceptor entry.ejb.Identifier at la/foton/infra/formatador/FormatadorServicoBean.execute ( V) id:51
============================================

Can you please change "find-entry-points" agent property to "true" for "Jboss" node and apply some load on business transaction under the application/server on which agent node is configured.

To set the find-entry-points agent property to true please find the below steps
- Please use the "To edit a registered node property" section in the following document to modify the value:
http://docs.appdynamics.com/display/PRO13S/App+Agent+Node+Properties#AppAgentNodeProperties-Toeditaregisterednodeproperty

The new value should take affect in couple of minutes, generate load on business transactions for few minutes.
And check whether you see any possible entry points discovered under BusinessTransactions.<latest-timestamp>.log under /opt/app/Tivoli/itm/appdynamics-agent/logs/ITMNode directory

Enabling find entry points ensures that stack traces are registered in logs so that we can give you some class names and method names so that you can create custom entry point rules. After that you can see the Business Transactions.

Please let us know how it goes.

Thanks,

Arun

Re: Jboss 7 monitoring only shows Memory data.

CommunityUser — Mon, 12 Aug 2013 18:36:41 GMT

hello!
I did everything you asked on the previous message, and it is still not working.
On a parallel line, I installed the agent on a local network application server, using Jboss 5, and it is working perfectly, all metrics ans backends are discovered.
Is there, maybe a limitation using SSH tunnel for the communication between agent and controller?

Re: Jboss 7 monitoring only shows Memory data.

Arun_Dasetty — Tue, 13 Aug 2013 09:13:23 GMT

Hi ,

Thanks for the update, we confirm the agent and controller cummincation should work fine through ssh tunnel, Can you confirm whether using actual domain name instead of the IP helps to resolved the agent connectivity errors. Also can you confirm whether you are able to ping the controller url without any auth credentials in wget/ping command.

We request you to make sure that agent/controller connectivity issues are resolved from your network end to debug further.

Thanks,

Arun

Re: Jboss 7 monitoring only shows Memory data.

Arun_Dasetty — Wed, 21 Aug 2013 08:49:29 GMT

Hi,

Adding to my earlier comments, we request you to make sure that the port 8090 is opened in ssh tunnel and also ssh tunnel is opened all the times as agent tries to connect to controller each minute and fails to send the data if there are intermediate network issues.

Do write back to us if the ssh tunnel connection is fine and agent have issues in connecting to controller, we will be gald to assist you further.

Thanks,

Arun

Re: Jboss 7 monitoring only shows Memory data.

CommunityUser — Wed, 21 Aug 2013 11:59:32 GMT

Hi Arun.

I made plenty of tests over the connection and it was fine.

After the JBOSS 5 application monitoring worked, I left jboss7 app behind to use the rest of my trial days studying the tool, which is very good.

Unfortunately my trial days are over and I can not work on that anymore, now we are planning to buy some licenses.

As soon as I get the licenses I´ll get back working on that.

Thanks for your help.

Re: Jboss 7 monitoring only shows Memory data.

Arun_Dasetty — Wed, 21 Aug 2013 13:48:06 GMT

Hi,

Thanks for the update, we understood your concern we are apologetic that we could not resolve the issue,we suspect this could happen only if no requests are made or the agent is configured in standalone startup script instead of domain.sh/bat file where apps are deployed in domain level and no apps are deployed/accessed on jvm with agent configure apart from network issues, also we still suspect it is network issue should be resolved to make sure there are no agent connectivity errors in agent.logs

We understood you could monitor jboss5 using agent in the same machine where jboss7 had issue and using the same controller, let us know if any disconnect, if the above said is right, we suspect configuring domain mode should help

We understood that your JBoss server is running in standalone mode, if the above said is wrong and your jboss server is running in domain mode, we request you to configure JBoss EAP server as follows:

1. in <jboss_install>/bin/domain.conf

change from
JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman"
change to
JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,com.appdynamics,com.appdynamics.,com.singularity,com.singularity."

2. in <jboss_install>/domain/configuration/domain.xml add jvm-options like below

<server-groups>
<server-group name="main-server-group" profile="full">
<jvm name="default">
<heap size="64m" max-size="512m"/>

<jvm-options>
<option value="-javaagent:<agent_install_dir>/javaagent.jar"/>
</jvm-options>

</jvm>

3. In <jboss_install>/domain/configuration/host.xml, change auto-start="true" as below and restart jboss server:
before change:
<server name="jc1agw01" group="agw-gen-server-group" auto-start="false">
after change:
<server name="jc1agw01" group="agw-gen-server-group" auto-start="true">

Please feel free to file new query for further assistance, we will be glad to assist you.

Thanks,

Arun