Mule ESB 3.5.2 Support

[CommunityUser]

Hi,

We are trying to configure AppDynamics agent for Mule ESB 3.5.2 version and we are seeing that we dont have any out of the box support for this version. We found from this link that starting 4.0.x version of AppD it supports Mule ESB 3.4 and previous releases.

https://docs.appdynamics.com/display/PRO40/Mule+ESB+Support

We are looking for options or alternatives to make the AppAgent work for us. Currently we are seeing that Machine Agent works and gives us stats but we don't see the AppAgent doing anything at all. Here is the configuration that we have in our wrapper.conf file in %MULE_HOME%/conf/wrapper.conf:

wrapper.java.additional.39=-Dcom.sun.management.jmxremote
wrapper.java.additional.40=-Dcom.sun.management.jmxremote.port=9426
wrapper.java.additional.41=-Dcom.sun.management.jmxremote.ssl=false
wrapper.java.additional.42=-Dcom.sun.management.jmxremote.authenticate=false
wrapper.java.additional.43=-Djava.rmi.server.hostname=localhost
wrapper.java.additional.44=-javaagent:"/opt/appdynamics/appagent/javaagent.jar
wrapper.java.additional.44.stripquotes=TRUE

Here is a snapshot of the servers that show up in our AppD console:

 

One more interesting thing here is that when I click on the UP icon of any of the node machine agent status, it indicates that App Agent is not installed. Please find the snapshot below:

When we do a service restart also, we only see that machine agent is working:

root      5649     1 82 23:14 pts/0    00:00:01 /opt/java/bin/java -jar /opt/appdynamics/machineagent

 Are we supposed to do anything for AppAgent? Is there a way to get this working? I tried reaching Mule ESB support folks but looks like this has to be done on AppD side itself for supporting their latest versions. Any help or alternatives to get App Agent working will be appreciated.

[CommunityUser]

I was looking at your config and you might have a typo. Instead of:

wrapper.java.additional.44=-javaagent:"/opt/appdynamics/appagent/javaagent.jar

Try:

wrapper.java.additional.44="-javaagent:/opt/appdynamics/appagent/javaagent.jar"

[CommunityUser]

Hi Frank,

I believe we used this config in wrapper.conf earlier

wrapper.java.additional.44=-javaagent:"/opt/appdynamics/appagent/javaagent.jar
wrapper.java.additional.44.stripquotes=TRUE

 Do you think I still need to use that stripquotes because as far as the server startup goes, I can see javaagent added to the startup arguments. I am not sure if this change made any difference because with earlier config, this is how my server startup looks like:

muleuser  8236  8234  0 Dec25 ?        00:15:40 /opt/jdk1.6.0_24/bin/java -Dmule.home=/opt/muleesb -Dmule.base=/opt/muleesb -Djava.net.preferIPv4Stack=TRUE -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:ErrorFile=/opt/muleesb/logs/err.log -Xloggc:/opt/muleesb/logs/gc.log -XX:+AlwaysPreTouch -server -XX:PermSize=128m -XX:MaxPermSize=256m -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60 -XX:+UseCMSInitiatingOccupancyOnly -Dmule.env=dev -XX:CompileThreshold=1000 -XX:MaxTenuringThreshold=8 -XX:TargetSurvivorRatio=90 -XX:SurvivorRatio=8 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=512m -XX:CMSFullGCsBeforeCompaction=1 -XX:CMSTriggerPermRatio=80 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSConcurrentMTEnabled -XX:+UseParNewGC -XX:+AggressiveOpts -Xss228k -Dmvel2.disable.jit=TRUE -Dmule.transport.http.singleDispatcherPerEndpoint=TRUE -Dmule.simpleLog=mule.log -DNAS_PROPERTY_HOME=/etc/stubhub/properties -Djetty.config.file=/opt/muleesb/apps/jetty-config.xml -Dmule.server.key.store.file=/opt/muleesb/conf/mulekeystore.jks -Dmule.client.trust.store.file=/opt/muleesb/conf/mulekeystore.jks -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9426 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -javaagent:/opt/appdynamics/appagent/javaagent.jar -XX:MaxPermSize=256m -Djava.endorsed.dirs=/opt/muleesb/lib/endorsed -Xms2048m -Xmx4096m -Djava.library.path=%LD_LIBRARY_PATH%:/opt/muleesb/lib/boot -classpath %MULE_LIB%:/opt/muleesb/conf:/opt/muleesb/lib/boot/commons-cli-1.2.jar:/opt/muleesb/lib/boot/commons-codec-1.3.jar:/opt/muleesb/lib/boot/jul-to-slf4j-1.6.1.jar:/opt/muleesb/lib/boot/licm-1.1.5.jar:/opt/muleesb/lib/boot/log4j-1.2.16.jar:/opt/muleesb/lib/boot/mule-module-boot-ee-3.5.2.jar:/opt/muleesb/lib/boot/mule-module-logging-3.5.2.jar:/opt/muleesb/lib/boot/oscore-2.2.4.jar:/opt/muleesb/lib/boot/propertyset-1.3.jar:/opt/muleesb/lib/boot/truelicense-1.29.jar:/opt/muleesb/lib/boot/truexml-1.29.jar:/opt/muleesb/lib/boot/wrapper-3.5.19.jar:/opt/muleesb/data-mapper -Dwrapper.key=ESE0FfDbiIj18SsbHwsSqzipPmj4l0Wu -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.disable_console_input=TRUE -Dwrapper.pid=8234 -Dwrapper.version=3.5.19-st -Dwrapper.native_library=wrapper -Dwrapper.arch=x86 -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 -Dwrapper.lang.domain=wrapper -Dwrapper.lang.folder=../lang org.mule.module.reboot.MuleContainerBootstrap restart0

 if you see this portion of the startup

-javaagent:/opt/appdynamics/appagent/javaagent.jar 

I see no different after changing the wrapper.conf to this

wrapper.java.additional.44="-javaagent:/opt/appdynamics/appagent/javaagent.jar"

 OR this:

wrapper.java.additional.44="-javaagent:/opt/appdynamics/appagent/javaagent.jar"
wrapper.java.additional.44.stripquotes=TRUE

I can see the startup showing the javaagent either way and I don't see any means to explicitly see if the appagent started.

Is there a way to see if the appagent started except from the console?

Is there a way to explicitly start appagent alone because i dont see logs for appagent but i do see logs for machineagent.

[Arun_Dasetty]

Hi,

Can you provide the following to debug furrher:

- controller-info.xml at /opt/appdynamics/appagent/conf 

- output of the following query issued where mule ESB JVM is up and running with -javagent entry added in wrapper file:

shell> ps -ef | grep "java"

We see appserver agent node itself does not come in UI would like to check whether agent started as part of JVM using ps grep output and if started fine whether agent config details controller-host, port, app, tier, node-name are provided in controller-info.xml file

- also attach the zipped version of logs and conf folder from /opt/appdynamics/appagent  location 

Once agent is up at UI and if no load discovered we will then define pojo rules on target if no ootb BTs are detected, Keep us posted how it goes.

Regards,

Arun

[CommunityUser]

Hi Arun,

Here is how the appagent/conf/controller-info.xml looks like in one of the nodes:

<?xml version="1.0" encoding="UTF-8"?>
<controller-info>
    <controller-host>srwd00col001.stubcorp.dev</controller-host>
    <controller-port>80</controller-port>
    <controller-ssl-enabled>false</controller-ssl-enabled>
    <enable-orchestration>false</enable-orchestration>
    <account-name></account-name>
    <account-access-key></account-access-key>
    <application-name>srwe02.com</application-name>
    <tier-name>int</tier-name>
    <node-name>srwe02int001</node-name>
    <force-agent-registration>false</force-agent-registration>
</controller-info>   

 I see no difference when we compare with machine agent controller-info.xml. I do see that appagent logs are empty whereas machineagent logs are present and I can see machine agent process running too. Here is the process info that you asked for:

muleuser 26644 26642  4 18:18 ?        00:00:58 /opt/jdk1.6.0_24/bin/java -Dmule.home=/opt/muleesb -Dmule.base=/opt/muleesb -Djava.net.preferIPv4Stack=TRUE -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:ErrorFile=/opt/muleesb/logs/err.log -Xloggc:/opt/muleesb/logs/gc.log -XX:+AlwaysPreTouch -server -XX:PermSize=128m -XX:MaxPermSize=256m -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60 -XX:+UseCMSInitiatingOccupancyOnly -Dmule.env=dev -XX:CompileThreshold=1000 -XX:MaxTenuringThreshold=8 -XX:TargetSurvivorRatio=90 -XX:SurvivorRatio=8 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=512m -XX:CMSFullGCsBeforeCompaction=1 -XX:CMSTriggerPermRatio=80 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSConcurrentMTEnabled -XX:+UseParNewGC -XX:+AggressiveOpts -Xss228k -Dmvel2.disable.jit=TRUE -Dmule.transport.http.singleDispatcherPerEndpoint=TRUE -Dmule.simpleLog=mule.log -DNAS_PROPERTY_HOME=/etc/stubhub/properties -Djetty.config.file=/opt/muleesb/apps/jetty-config.xml -Dmule.server.key.store.file=/opt/muleesb/conf/mulekeystore.jks -Dmule.client.trust.store.file=/opt/muleesb/conf/mulekeystore.jks -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9426 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -javaagent:/opt/appdynamics/appagent/javaagent.jar -XX:MaxPermSize=256m -Djava.endorsed.dirs=/opt/muleesb/lib/endorsed -Xms2048m -Xmx4096m -Djava.library.path=%LD_LIBRARY_PATH%:/opt/muleesb/lib/boot -classpath %MULE_LIB%:/opt/muleesb/conf:/opt/muleesb/lib/boot/commons-cli-1.2.jar:/opt/muleesb/lib/boot/commons-codec-1.3.jar:/opt/muleesb/lib/boot/jul-to-slf4j-1.6.1.jar:/opt/muleesb/lib/boot/licm-1.1.5.jar:/opt/muleesb/lib/boot/log4j-1.2.16.jar:/opt/muleesb/lib/boot/mule-module-boot-ee-3.5.2.jar:/opt/muleesb/lib/boot/mule-module-logging-3.5.2.jar:/opt/muleesb/lib/boot/oscore-2.2.4.jar:/opt/muleesb/lib/boot/propertyset-1.3.jar:/opt/muleesb/lib/boot/truelicense-1.29.jar:/opt/muleesb/lib/boot/truexml-1.29.jar:/opt/muleesb/lib/boot/wrapper-3.5.19.jar:/opt/muleesb/data-mapper -Dwrapper.key=cXOl5Taf8wron-Dev-KguA700sbWK2WR -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.disable_console_input=TRUE -Dwrapper.pid=26642 -Dwrapper.version=3.5.19-st -Dwrapper.native_library=wrapper -Dwrapper.arch=x86 -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 -Dwrapper.lang.domain=wrapper -Dwrapper.lang.folder=../lang org.mule.module.reboot.MuleContainerBootstrap start0

root     27366     1  0 18:23 pts/0    00:00:01 /opt/java/bin/java -jar /opt/appdynamics/machineagent/machineagent.jar

 I dont see any explicit process that is indicating the appagent being running on that node. Is there a process to run appagent explicitly? Do i need to do anything seperately for appagent?

[CommunityUser · Accepted solution]

Try removing the quotes, and the entire stripquotes line to see if that helps. Also, make sure nowhere else in your config is there a 

wrapper.java.additional.44

The number needs to be unique for every config else it will ingnore the entry, but I think you already know all of this and your config looks good.

Also, is the folder /opt/appdynamics/appagent/ and all of the files within it executable and owned by the same user that is launching MuleESB? Sometimes permissions issues can prevent the agent from launching and/or writting logs.

[CommunityUser]

Thank you Frank for a quick response. I did see that issue is with user and permissions on /opt/appdynamics folder being the root cause. I have corrected the user permissions and now both appdynamics and muleESB start as same user and that helped in starting the appAgent in first place.