Jboss 7 monitoring only shows Memory data.

[CommunityUser]

I have followed the manual steps to configure monitoring on a JBOSS 7.1.1, after all steps, my controller only shows Memory and garbage collector metrics for the monitored JBOSS instance.

Have you ever seen this before? What can it be?

[CommunityUser]

Below are the java versions if it helps to find what is happening.

java version "1.7.0_25"
Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)

[Arun_Dasetty]

Hi,

Thanks for writing in. We confirm we have customer monitoring JBoss 7 with successfully monitoring business transactions under BT with call stats fine under node dashboard -> dashbpard as opposed to screenshot "app1.JPG"

Can you please provide the following details:
- we see such issues if agent is disabled under node dashboard -> agents -> "appserver agent" screen, we assume that agent is enabled under "appserver agent" section in reference to screenshot "app3.JPG" in this case, let us know if any disconnect.
- Can you confirm whether you have generated load on the jboss server (did you make any business transaction or requests like servlets, jms etc ) on the server with agent
- Can you please send the zipped version logs folder under <agent_install_dir>/logs, we would like to check if any errors realted to BT detection

Please keep us posted the requested details.

Thanks,

Arun

[CommunityUser]

Requested logs follows attached.

Thanks a lot

[Arun_Dasetty]

Hi ,

We see that you have sent controller server.log, we request you to send logs under <agent_install_dir>/logs, also you can select "all logs in the agent directory" option (the first option in attached screenshot) from jboss node directory

we request you to revert with logs under agent directory, where agent dir is the path you have confiugred for -javaagent: parameter in jboss startup script.

Thanks,

Arun

[CommunityUser]

Ok, our server is down right now, I´ll send you the correct logs as soon as it is available to me.

Thank you.

[Arun_Dasetty]

Hi,

Thanks for the update. sure keep us posted the requested logs.

Thanks,

Arun

[CommunityUser]

Follows attached the agent logs.

thanks.

[Arun_Dasetty]

Hi,

Thanks for posting back, we see different errors in agent.log, and we see the connection refused error repetively in all agent.logs:
----------------------
[AD Thread Pool-Global1] 08 Aug 2013 09:38:46,225 ERROR ConfigurationChannel - Fatal transport error: Connection refused
[AD Thread Pool-Global1] 08 Aug 2013 09:38:46,225  WARN ConfigurationChannel - Invalid response for configuration request from controller/could not connect to controller. Msg : Fatal transport error: Connection refused
..
[AD Thread Pool-Global0] 08 Aug 2013 09:39:07,415 ERROR ConfigurationChannel - Fatal transport error: Connection refused
[AD Thread Pool-Global0] 08 Aug 2013 09:39:07,415  WARN ConfigurationChannel - Could not connect to the controller/invalid response from controller, cannot get initialization information, controller host [127.0.0.1], port[8090], exception [Fatal transport error: Connection refused]
-----------------------

we would like to inform you that, It is not recommended to provide controller host name as 127.0.0.1,

- Can you confirm whether controller is running in the same host wheher jboss server with agent is running

Can you confirm whether controller is accessible currently?

- Is your jboss server is running domain mode or standalone mode and what is the jboss version?

- Can you confirm whether you have configured jboss seever as per docs, if the above said is right can you provide the script/command you are using to start jboss server?
http://docs.appdynamics.com/display/PRO13S/JBoss+Startup+Settings#JBossStartupSettings-Toaddthejavaa...

We requested the above information as we see two different agent logs with different jvm process id from which we suspect that agent is starting twice or not configured on domain with application deployed.
-----------------------------------------------------------------
from agent.2013_08_08__09_17_04.0.log:
[Thread-0] 08 Aug 2013 08:45:23,085  INFO ConfigurationChannel - JVM PID: 6127
[AD Thread-Metric Reporter0] 08 Aug 2013 09:39:02,055  WARN MetricHandler - Metric Reporter Queue full. Dropping metrics.

from agent.2013_08_08__08_45_21.0.log
[Thread-0] 08 Aug 2013 08:45:21,403  INFO AgentKernel - Starting Java Agent at Thu Aug 08 08:45:21 BRT 2013 ...
[Thread-0] 08 Aug 2013 09:17:07,036  INFO ConfigurationChannel - JVM PID: 6492
[AD Thread Pool-Global1] 08 Aug 2013 09:38:23,598 ERROR ConfigurationChannel - Fatal transport error: Connection refused

----------------------------------------------------------------

Thanks,

Arun

[CommunityUser]

Hi Akumar.

The monitored server is connected to the controler via reverse SSH tunnel, those connection errors ocurred because the reverse tunnel was not enabled, when I enable it, the message stops. here is the reverse ssh tunnel command (ssh -vvv -nNT -R 8090:127.0.0.1:8090 blkhmlbrb@172.23.2.142) thats why the server is using 127.0.0.1 as controller IP.

Jboss is version 7 and running on standalone mode.

standalone.sh follows attached.

[Arun_Dasetty]

Hi,

Thanks for posting back, we see that you are using jboss-as-7.1.1 version, and can you you please confirm whether you have configure standalone.conf and have added the following entry specific to jboss 7.1.1 version as per docs:

For JBoss 7.1.1, add the additional packages as shown here:

JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,com.appdynamics,com.appdynamics.,com.singularity,com.singularity.,org.jboss.logmanager"

-

http://docs.appdynamics.com/display/PRO13S/JBoss+Startup+Settings#JBossStartupSettings-Toaddthejavaa...

and also make sure that you have configured as per steps 1,2,3 as per docs and if you still do not see any BTs discovered after making load on jboss server, can you confirm whether you the URL is hitting is servlet request, as the static html requests will not be monitored by agent, if the issue still persists send us the latest agent logs and standalone.conf

Thanks,

Arun

[CommunityUser]

Agent Logs attached.

only those 2 files had changes today.

All the others are still modified dated as friday

[Arun_Dasetty]

Hi,

Thanks for posting back and providing logs, Can you please try as follows:

- we still see the errors which must be resolved as this affect the reporting of agent metrics, we understood that you have mentioned that you have set controler via reverse SSH tunnel, but we are expecting that the following errors should be resolved and should not occure recursively in agent.log:

-----------------
[AD Thread-Metric Reporter0] 12 Aug 2013 08:48:48,325 ERROR MetricHandler - Error sending metrics - will requeue for later transmission
com.singularity.ee.agent.commonservices.metricgeneration.metrics.g: Fatal transport error: Connection refused
-----------

- Can you remove the following option from jvm_opts "-Dorg.jboss.boot.log.file=/var/foton/blkhmlbrb/logs/boot.log " and restart jboss

- We see the following entry points applied successfully from BCI.log:
=================================
    Line 5858:         Applying method interceptor entry.Servlet at org/jboss/resteasy/plugins/server/servlet/HttpServletDispatcher.service ((Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse; ) V) id:3
    Line 8321:         Applying method interceptor entry.webservice.jboss.ws.impl.Service at org/jboss/ws/core/server/ServiceEndpointInvoker.invoke ((Lorg/jboss/wsf/spi/invocation/InvocationContext; ) V ) id:7
    Line 9381:         Applying method interceptor entry.Servlet at org/jboss/as/web/WelcomeContextConsoleServlet.service ((Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse; ) V) id:8
    Line 12869:         Applying method interceptor entry.ejb.Identifier at la/foton/infra/formatador/FormatadorServicoBean.execute (  V) id:51
============================================

Can you please change "find-entry-points" agent property to "true" for "Jboss" node and apply some load on business transaction under the application/server on which agent node is configured.

To set the find-entry-points agent property to true please find the below steps
- Please use the "To edit a registered node property" section in the following document to modify the value:
http://docs.appdynamics.com/display/PRO13S/App+Agent+Node+Properties#AppAgentNodeProperties-Toeditar...

The new value should take affect in couple of minutes, generate load on business transactions for few minutes.
And check whether you see any possible entry points discovered under BusinessTransactions.<latest-timestamp>.log under /opt/app/Tivoli/itm/appdynamics-agent/logs/ITMNode directory

Enabling find entry points ensures that stack traces are registered in logs so that we can give you some class names and method names so that you can create custom entry point rules. After that you can see the Business Transactions.

Please let us know how it goes.

Thanks,

Arun

[CommunityUser]

hello!
I did everything you asked on the previous message, and it is still not working.
On a parallel line, I installed the agent on a local network application server, using Jboss 5, and it is working perfectly, all metrics ans backends are discovered.
Is there, maybe a limitation using SSH tunnel for the communication between agent and controller?

[Arun_Dasetty]

Hi ,

Thanks for the update, we confirm the agent and controller cummincation should work fine through ssh tunnel, Can you confirm whether using  actual domain name instead of the IP helps to resolved the agent connectivity errors. Also can you confirm whether you are able to ping the controller url without any auth credentials in wget/ping command.

We request you to make sure that agent/controller connectivity issues are resolved from your network end to debug further.

Thanks,

Arun

[Arun_Dasetty]

Hi,

Adding to my earlier comments, we request you to make sure that the port 8090 is opened in ssh tunnel and also ssh tunnel is opened all the times as agent tries to connect to controller each minute and fails to send the data if there are intermediate network issues.

Do write back to us if the ssh tunnel connection is fine and agent have issues in connecting to controller, we will be gald to assist you further.

Thanks,

Arun

[CommunityUser]

Hi Arun.

I made plenty of tests over the connection and it was fine.

After the JBOSS 5 application monitoring worked, I left jboss7 app behind to use the rest of my trial days studying the tool, which is very good.

Unfortunately my trial days are over and I can not work on that anymore, now we are planning to buy some licenses.

As soon as I get the licenses I´ll get back working on that.

Thanks for your help.

[Arun_Dasetty]

Hi,

Thanks for the update, we understood your concern we are apologetic that we could not resolve the issue,we suspect this could happen only if no requests are made or the agent is configured in standalone startup script instead of domain.sh/bat file where apps are deployed in domain level and no apps are deployed/accessed on jvm with agent configure apart from network issues, also we still suspect it is network issue should be resolved to make sure there are no agent connectivity errors in agent.logs

We understood you could monitor jboss5 using agent in the same machine where jboss7 had issue and using the same controller, let us know if any disconnect, if the above said is right, we suspect configuring domain mode should help

We understood that your JBoss server is running in standalone mode, if the above said is wrong and your jboss server is running in domain mode, we request you to configure JBoss EAP server as follows:

1. in <jboss_install>/bin/domain.conf

change from
JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman"
change to
JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,com.appdynamics,com.appdynamics.,com.singularity,com.singularity."

2. in <jboss_install>/domain/configuration/domain.xml add jvm-options like below

<server-groups>
<server-group name="main-server-group" profile="full">
<jvm name="default">
<heap size="64m" max-size="512m"/>
<!-- new changes here -->
<jvm-options>
<option value="-javaagent:<agent_install_dir>/javaagent.jar"/>
</jvm-options>
<!-- new changes end here -->
</jvm>

3. In <jboss_install>/domain/configuration/host.xml, change auto-start="true" as below and restart jboss server:
before change:
<server name="jc1agw01" group="agw-gen-server-group" auto-start="false">
after change:
<server name="jc1agw01" group="agw-gen-server-group" auto-start="true">

Please feel free to file new query for further assistance, we will be glad to assist you.

Thanks,

Arun

[CommunityUser]

Here is the java app running.

[blkhmlbrb@hib340210 share]$ ps -ef |grep java
514 3734 3676 0 Aug09 ? 00:29:00 /usr/java/jdk1.7.0_25/bin/java -D[Standalone] -XX:+UseCompressedOops -XX:+TieredCompilation -server -Dprog.name=BRBBanklinkJboss -Xms2048m -Xmx2048m -XX:MaxPermSize=512M -Djboss.server.log.dir=/home/blkhmlbrb/jboss-deploy/logs/ -Dfile.encoding=ISO-8859-1 -Dlog4j.debug=true -Djava.awt.headless=true -XX:+UseParallelGC -XX:SurvivorRatio=6 -Djboss.modules.system.pkgs=org.jboss.byteman,com.appdynamics,com.appdynamics.,com.singularity,com.singularity.,org.jboss.logmanager -Djava.awt.headless=true -XX:-UseSplitVerifier -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:/usr/share/jboss-as/jboss-as-7.1.1.Final/modules/org/jboss/logmanager/main/jboss-logmanager-1.2.2.GA.jar:/usr/share/jboss-as/jboss-as-7.1.1.Final/modules/org/jboss/logmanager/log4j/main/jboss-logmanager-log4j-1.0.0.GA.jar:/usr/share/jboss-as/jboss-as-7.1.1.Final/modules/org/apache/log4j/main/log4j-1.2.16.jar -javaagent:/home/blkhmlbrb/apps/AppServerAgent/javaagent.jar -Dorg.jboss.boot.log.file=/var/foton/blkhmlbrb/logs/boot.log -Dlogging.configuration=file:/usr/share/jboss-as/jboss-as-7.1.1.Final/standalone/configuration/logging.properties -jar /usr/share/jboss-as/jboss-as-7.1.1.Final/jboss-modules.jar -mp /usr/share/jboss-as/jboss-as-7.1.1.Final/modules -jaxpmodule javax.xml.jaxp-provider -logmodule org.jboss.logmanager.LogManager org.jboss.as.standalone -Djboss.home.dir=/usr/share/jboss-as/jboss-as-7.1.1.Final -b 0.0.0.0

[CommunityUser]

Hello!

Thanks for your reply.

I did all the steps you mentioned on your message, standalone.conf is attached.

Yes there are transactions going on the server, including lots of oracle database queries as I could see on the APP Server Log.