agent can't start on Websphere - log class cannot be found or is not useable

[CommunityUser]

Hello,

This has ever happened on viewer - after reinstall, the viewer works - see previous post.

But this time, happens on agent instead of viewer, and on Websphere (on tomcat no problem).

The log as

[main] 11 Nov 2013 12:47:22,848  INFO JavaAgent - ####################################################################################
[main] 11 Nov 2013 12:47:22,848  INFO JavaAgent - Agent Directory [C:\jdk6\agent]
[main] 11 Nov 2013 12:47:22,848  INFO JavaAgent - Using Agent Version [Server Agent Lite v3.3.4.0 RC Build Date 2013-03-20 20:23]
[main] 11 Nov 2013 12:47:22,848  INFO AgentKernel - Starting Agent at Mon Nov 11 12:47:22 EST 2013 ...
[main] 11 Nov 2013 12:47:22,848  INFO AgentKernel - JVM Runtime:
    java.home=C:\Program Files\IBM\SDP\runtimes\base_v7\java\jre
    java.vm.vendor=IBM Corporation
    java.vm.name=IBM J9 VM
    java.runtime.version=pwi3260sr13fp2ifx-20130513_01 (SR13 FP2)
[main] 11 Nov 2013 12:47:22,848  INFO AgentKernel - OS Runtime:
    os.name=Windows XP
    os.arch=x86
    os.version=5.1 build 2600 Service Pack 3
[main] 11 Nov 2013 12:47:22,848  INFO AgentKernel - JVM Args : -Xjcl:jclscar_24 | -Dcom.ibm.oti.vm.bootstrap.library.path=C:\Program Files\IBM\SDP\runtimes\base_v7\java\jre\bin | -Dsun.boot.library.path=C:\Program Files\IBM\SDP\runtimes\base_v7\java\jre\bin | -Djava.library.path=C:\Program Files\IBM\SDP\runtimes\base_v7\java\jre\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files\IBM\SDP\runtimes\base_v7\bin;C:\Program Files\IBM\SDP\runtimes\base_v7\java\bin;C:\Program Files\IBM\SDP\runtimes\base_v7\java\jre\bin;C:\Program Files\Attachmate\EXTRA!\;. | -Djava.home=C:\Program Files\IBM\SDP\runtimes\base_v7\java\jre | -Djava.ext.dirs=C:\Program Files\IBM\SDP\runtimes\base_v7\java\jre\lib\ext | -Duser.dir=C:\workspace\base_v7\t1 | -Djava.runtime.version=pwi3260sr13fp2ifx-20130513_01 (SR13 FP2) | -Xdump | -Dconsole.encoding=Cp850 | -Djava.class.path=C:\workspace\base_v7\t1\properties;C:\Program Files\IBM\SDP\runtimes\base_v7\properties;C:\Program Files\IBM\SDP\runtimes\base_v7\lib\startup.jar;C:\Program Files\IBM\SDP\runtimes\base_v7\lib\bootstrap.jar;C:\Program Files\IBM\SDP\runtimes\base_v7/lib/lmproxy.jar;C:\Program Files\IBM\SDP\runtimes\base_v7/lib/urlprotocols.jar;C:\Program Files\IBM\SDP\runtimes\base_v7\java\lib\tools.jar | -Declipse.security | -Dosgi.install.area=C:\Program Files\IBM\SDP\runtimes\base_v7 | -Dosgi.configuration.area=C:\workspace\base_v7\t1/configuration | -Dosgi.framework.extensions=com.ibm.cds,com.ibm.ws.eclipse.adaptors | -Xshareclasses:name=webspherev70,nonFatal | -Xscmx50M | -Dsun.reflect.inflationThreshold=250 | -Xbootclasspath/p:C:\Program Files\IBM\SDP\runtimes\base_v7/java/jre/lib/ext/ibmorb.jar;C:\Program Files\IBM\SDP\runtimes\base_v7/java/jre/lib/ext/ibmext.jar | -Djava.class.path=C:\workspace\base_v7\t1/properties;C:\Program Files\IBM\SDP\runtimes\base_v7/properties;C:\Program Files\IBM\SDP\runtimes\base_v7/lib/startup.jar;C:\Program Files\IBM\SDP\runtimes\base_v7/lib/bootstrap.jar;C:\Program Files\IBM\SDP\runtimes\base_v7/lib/jsf-nls.jar;C:\Program Files\IBM\SDP\runtimes\base_v7/lib/lmproxy.jar;C:\Program Files\IBM\SDP\runtimes\base_v7/lib/urlprotocols.jar;C:\Program Files\IBM\SDP\runtimes\base_v7/deploytool/itp/batchboot.jar;C:\Program Files\IBM\SDP\runtimes\base_v7/deploytool/itp/batch2.jar;C:\Program Files\IBM\SDP\runtimes\base_v7/java/lib/tools.jar;C:\workspace\config; | -Dibm.websphere.internalClassAccessMode=allow | -Xquickstart | -Xverify:none | -Dws.ext.dirs=C:\Program Files\IBM\SDP\runtimes\base_v7/java/lib;C:\workspace\base_v7\t1/classes;C:\Program Files\IBM\SDP\runtimes\base_v7/classes;C:\Program Files\IBM\SDP\runtimes\base_v7/lib;C:\Program Files\IBM\SDP\runtimes\base_v7/installedChannels;C:\Program Files\IBM\SDP\runtimes\base_v7/lib/ext;C:\Program Files\IBM\SDP\runtimes\base_v7/web/help;C:\Program Files\IBM\SDP\runtimes\base_v7/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime | -Dderby.system.home=C:\Program Files\IBM\SDP\runtimes\base_v7/derby | -Dcom.ibm.itp.location=C:\Program Files\IBM\SDP\runtimes\base_v7/bin | -Djava.util.logging.configureByServer=true | -Duser.install.root=C:\workspace\base_v7\t1 | -Djavax.management.builder.initial=com.ibm.ws.management.PlatformMBeanServerBuilder | -Dwas.install.root=C:\Program Files\IBM\SDP\runtimes\base_v7 | -Dpython.cachedir=C:\workspace\base_v7\t1/temp/cachedir | -Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager | -Dserver.root=C:\workspace\base_v7\t1 | -Dcom.ibm.security.jgss.debug=off | -Dcom.ibm.security.krb5.Krb5Debug=off | -Dcom.ibm.websphere.orb.uniqueServerName=true | -Djdbc_driver=oracle.jdbc.driver.OracleDriver | -Doracle.net.tns_admin=/workspace/config/admin | -Dcom.ibm.ws.management.event.pull_notification_timeout=120000 | -Xms1024M | -Xmx1024M | -javaagent:C:\jdk6\agent\javaagent.jar | -Djava.security.policy=C:\workspace\base_v7\t1/properties/server.policy | -Dsun.java.command=com.ibm.wsspi.bootstrap.WSPreLauncher -nosplash -application com.ibm.ws.bootstrap.WSLauncher com.ibm.ws.runtime.WsServer C:\workspace\base_v7\t1\config | -Dsun.java.launcher=SUN_STANDARD |
[main] 11 Nov 2013 12:47:22,848  INFO AgentKernel - Running obfuscated agent
[main] 11 Nov 2013 12:47:22,895  INFO XMLConfigManager - Default Controller Info Resolver is running
[main] 11 Nov 2013 12:47:22,895  INFO XMLConfigManager - Default Controller Info Resolver using system property [appdynamics.controller.hostName] for controller host name [127.0.0.1]
[main] 11 Nov 2013 12:47:22,895  INFO XMLConfigManager - Default Controller Info Resolver using system property [appdynamics.controller.port] for controller port [8990]
[main] 11 Nov 2013 12:47:22,895  INFO XMLConfigManager - Default Controller Info Resolver finished running
[main] 11 Nov 2013 12:47:22,895  INFO XMLConfigManager - Default Host Identifier Resolver is running
[main] 11 Nov 2013 12:47:22,895  INFO XMLConfigManager - Default Host Identifier Resolver using host name for unique host identifier [16DHG2UA2060PQH]
[main] 11 Nov 2013 12:47:22,895  INFO XMLConfigManager - Default Host Identifier Resolver finished running
[main] 11 Nov 2013 12:47:22,895  INFO XMLConfigManager - Default Agent Registration Info Resolver is running
[main] 11 Nov 2013 12:47:22,895  INFO XMLConfigManager - Default Agent Registration Info Resolver using system property [appdynamics.agent.applicationName] for application name [Server App]
[main] 11 Nov 2013 12:47:22,895  INFO XMLConfigManager - Default Agent Registration Info Resolver using system property [appdynamics.agent.tierName] for tier name [Server Tier]
[main] 11 Nov 2013 12:47:22,895  INFO XMLConfigManager - Default Agent Registration Info Resolver using system property [appdynamics.agent.nodeName] for node name [Server Node]
[main] 11 Nov 2013 12:47:22,895  INFO XMLConfigManager - Default Agent Registration Info Resolver finished running
[main] 11 Nov 2013 12:47:22,895  INFO XMLConfigManager - Default Agent Account Info Resolver is running
[main] 11 Nov 2013 12:47:22,895  INFO XMLConfigManager - Default Agent Account Info Resolver using system property [appdynamics.agent.accountName] for account name [customer1]
[main] 11 Nov 2013 12:47:22,895  INFO XMLConfigManager - Default Agent Account Info Resolver using system property [appdynamics.agent.accountAccessKey] for account access key [customer1]
[main] 11 Nov 2013 12:47:22,895  INFO XMLConfigManager - Default Agent Account Info Resolver finished running
[main] 11 Nov 2013 12:47:22,910 ERROR JavaAgent - Could Not Start Java Agent
java.lang.ExceptionInInitializerError
    at java.lang.J9VMInternals.initialize(J9VMInternals.java:222)
    at java.lang.J9VMInternals.initialize(J9VMInternals.java:167)
    at java.lang.J9VMInternals.initialize(J9VMInternals.java:167)
    at com.singularity.ee.util.httpclient.d.b(d.java:107)
    at com.singularity.ee.util.httpclient.d.<clinit>(d.java:40)
    at java.lang.J9VMInternals.initializeImpl(Native Method)
    at java.lang.J9VMInternals.initialize(J9VMInternals.java:200)
    at com.singularity.ee.rest.controller.request.ib.a(ib.java:172)
    at com.singularity.ee.agent.appagent.kernel.config.xml.e.b(e.java:160)
    at com.singularity.ee.agent.appagent.kernel.config.xml.e.<init>(e.java:103)
    at com.singularity.ee.agent.appagent.kernel.config.b.a(b.java:20)
    at com.singularity.ee.agent.appagent.kernel.b.h(b.java:91)
    at com.singularity.ee.agent.appagent.kernel.JavaAgent.initialize(JavaAgent.java:116)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:600)
    at com.singularity.ee.agent.appagent.AgentEntryPoint.premain(AgentEntryPoint.java:153)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:600)
    at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:323)
    at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:338)
Caused by: org.apache.commons.logging.LogConfigurationException: User-specified log class 'com.singularity.ee.agent.pb' cannot be found or is not useable.
    at org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:798)
    at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:601)
    at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:333)
    at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:307)
    at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:645)
    at org.apache.commons.httpclient.params.DefaultHttpParams.<clinit>(DefaultHttpParams.java:54)
    at java.lang.J9VMInternals.initializeImpl(Native Method)
    at java.lang.J9VMInternals.initialize(J9VMInternals.java:200)
    ... 23 more

Questions:

1. why log class not found or not useable?

2. how to change the log class?

Thanks.

John

[Arun_Dasetty]

Hi John,

We understood your concern, Can you please check the integrity of download appserveragentlite archive and we request you to download again and doing a fresh installation and see how it goes, if you still see the issue, we request you to send the server version details and complete zipped version of agent logs and error stack trace to assist you better?

Also let us know if there are any log4j libs in classpath we see such issues when there are conflicting libs in classpath.

Thanks,

Arun

[CommunityUser]

Hi Arun,

I re-did everything (clean, download, install, restart, etc) - got same result.

download version:

AppDynamics Lite Agent Installation (version: 3.3.4.0) - I tried the IBM one and non-IBM one, same result

Minimum Java Version for Viewer - 1.5
How to start the Viewer

-------------------------
java -jar adlite-viewer.jar

IBM Websphere 7.0.29 - jdk

C:\Program Files\IBM\SDP\runtimes\base_v7\java\bin>java -version
java version "1.6.0"
Java(TM) SE Runtime Environment (build pwi3260sr13fp2ifix-20130513_01(SR13 FP2+197722))
IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Windows XP x86-32 jvmwi3260sr13fp2-20130423_146146 (JIT enabled, AOT enabled)
J9VM - 20130423_146146
JIT  - r9_20130108_31100ifx1
GC   - 20121212_AA)
JCL  - 20130513_01

jdk for agent

C:\>java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) Client VM (build 20.45-b01, mixed mode, sharing)

Websphere config folder contain the following log for application

log4j-1.2.7.jar

The log file for agent is same as post last time - only the timestamp difference.

log for viewer as

C:\jdk6\viewer>java -jar adlite-viewer.jar
Starting ... AppDynamics Lite Viewer 2.1.1 GA Build Date 2012-08-05 21:13:55
Lite Viewer Home Directory[C:\jdk6\viewer]
Using Java Version [1.6] for Lite Viewer
Starting adlite-viewer on port 8990.
2013-11-12 09:36:31.198::INFO:  Logging to STDERR via org.mortbay.log.StdErrLog
2013-11-12 09:36:31.214::INFO:  jetty-
2013-11-12 09:36:31.527::INFO:  Extract jar:file:/C:/jdk6/viewer/adlite-viewer.jar!/ to c:\tmp\Jetty_0_0_0_0_8990_adlite-viewer.jar____-x5l502\webapp
2013-11-12 09:36:36.089::INFO:  NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
Logging Configuration File[C:\jdk6\viewer\log4j.xml]
Nov 12, 2013 9:36:36 AM com.appdynamics.lite.viewer.core.o e
INFO: initializing rules ............
2013-11-12 09:36:37.042::INFO:  Started SocketConnector @ 0.0.0.0:8990
Type 'shutdown' to shutdown the viewer process


 Viewer Lite Shell:>

That's all info I could provide.

It's the log factory causing agent unable to start.

Thanks.

John

[Arun_Dasetty]

Hi John,

Can you try the following:

- create fresh appserver profile in websphere and try configuring AppServerAgentLite-ibm.zip avaialble in Lite download artifact?

- restart the new appserver profile and see whether you still see the issue?

If the issue persists please provide the following:

- zipped version AppServerAgent Lite installation folder that you used to configure -javaagent for ibm websphere server, we would like try using same artifact

- screenshot of ibm websphere UI or script file in which you configured -javagent entry

let us know how it goes

Thanks,

Arun

[CommunityUser]

Hello Arun.

The problem still exists.

I downloaded again (this week, the 3rd time to down) from 

http://info.appdynamics.com/free_java_profiler_monitoring_tool.html?utm_term=appdynamics|e|g&utm_med...

All downloaded files almost same (except timestamp different). Then I installed agent (attached), created blank new Websphere profile (console setting attached). part of the server.xml as

 <jvmEntries debugArgs="" debugMode="false" disableJIT="false" executableJarFileName="" genericJvmArguments="-Xquickstart -javaagent:C:\jdk6\agent\javaagent.jar" hprofArguments="" runHProf="false" verboseModeClass="false" verboseModeGarbageCollection="false" verboseModeJNI="false" xmi:id="JavaVirtualMachine_1183122130078">

Websphere server Base 7.0.0.29 log attached.

Normally in command line, we can use -D to set the log factory.

Anyway, now all blank new setting, problem still persists.

Please advise.

John

[Arun_Dasetty]

Hi John,

We have tried in local using same agent provided on IBM WAS 7.x server, Unfortunately we could not replicate the issue(refer attached test logs for reference), Can you please try the following and let us know if that makes any difference:

- Move the agent install directory to different drive or directory path , outside of  C:\jdk6 path

- remove -Xquickstart in "genericJVMargs" text fiedl in ibm config screen and restart the server

If the issue still persist can you try the following agent artifact instead and let us know how it goes

Thanks,

Arun

[CommunityUser]

Hello Arun,

I did

1. removed all agent/folder

2. installed the one you post in c:\tmp folder

3. removed -Xquickstart in "genericJVMargs"

4. change to

-javaagent:C:\tmp\agent\javaagent.jar

5. created new Webpshere profile (attached).

however, problem still exists.

can you please try my profile and advise?

Thanks

John

[Arun_Dasetty]

Hi John,

Thanks for the logs, Unfortunately we cannot start profile download from one machine IBM installation to another machine IBM server , unlike we can do in tomcat server where the same tomcat on machine1 will work for machine2 saying they are potable.

We suspect this issue is specific with Lite agent and websphere version though 7.x is listed, it could happen that particular version have issue, Can you please let us know if trying with AD PRO free version is an option here, as we could not reproduce the issue using Lite agent on ibm websphere 7.0.x version (not the same minor version)

http://info.appdynamics.com/30-day-trial.html

Please let us know how it goes using AD PRO agent and controller.

Thanks,

Arun

[CommunityUser]

Hi Arun,

That pro version is working as expected - that's good.

However, I'm wondering, how to separate the agent from it? as you know, I'm just interested in the agent, and we don't want to use the  commercial version yet and it's too heavy and we already installed other monitor tools.

Thanks.

John