Problems starting app server agent on IBM WebSphere Application Server

[Hans_Mueller]

I have already successfully appdynamics app server agent on seven application servers of my WebSphere network deployment environment. Unfortunately, the setup on the last server I could not successfully finish my setup. If I set the following parameter on my JVM:

javaagent:/opt/IBM/AppAgents/AppServerAgent/javaagent.jar -Dappdynamics.agent.tierName=<placeholde> -Dappdynamics.agent.nodeName=<placeholder>

I see the following error on the log of the app server agent:

[AD Agent init] 09 Feb 2022 16:11:21,251 INFO ConfigurationChannel - Container id retrieval enabled: true
[AD Agent init] 09 Feb 2022 16:11:21,252 WARN ConfigurationChannel - Unable to use /proc/self/cgroup for unique hostname, could not locate container ID
[AD Agent init] 09 Feb 2022 16:11:21,252 INFO ConfigurationChannel - Agent node meta-info thus far: ProcessID;111140;appdynamics.ip.addresses;21.0.11.44,21.1.11.44;appdynamicsHostName;xxxx
[AD Agent init] 09 Feb 2022 16:11:21,252 INFO ConfigurationChannel - Detected node meta info: [Name:ProcessID, Value:111140, Name:appdynamics.ip.addresses, Value:xxxx,xxxx, Name:appdynamicsHostName, Value:xxxx, Name:supportsDevMode, Value:true]
[AD Agent init] 09 Feb 2022 16:11:21,252 INFO ConfigurationChannel - Sending Registration request with: Application Name [xxxx], Tier Name [xxxx], Node Name [xxxx], Host Name [xxxx] Node Unique Local ID [xxxx], Version [Server Agent #21.11.3.33314 v21.11.3 GA compatible with 4.4.1.0 r1f94344f9fd88fc14fe39a33494b03e4bb555a6d release/21.11.0]
[AD Agent init] 09 Feb 2022 16:11:21,309 WARN AgentErrorProcessor - Agent error occurred, [name,transformId]=[com.singularity.XMLConfigManager - javax.xml.parsers.FactoryConfigurationError,2147483647]
[AD Agent init] 09 Feb 2022 16:11:21,309 WARN AgentErrorProcessor - 4 instance(s) remaining before error log is silenced
[AD Agent init] 09 Feb 2022 16:11:21,309 WARN XMLConfigManager - Error refreshing agent configuration
javax.xml.parsers.FactoryConfigurationError: Provider javax.xml.parsers.DocumentBuilderFactory could not be instantiated: java.util.ServiceConfigurationError: javax.xml.parsers.DocumentBuilderFactory: Provider org.apache.xerces.jaxp.DocumentBuilderFactoryImpl not a subtype
at javax.xml.parsers.DocumentBuilderFactory.newInstance(Unknown Source) ~[?:?]
at java.util.prefs.XmlSupport.loadPrefsDoc(XmlSupport.java:252) ~[?:1.8.0]
at java.util.prefs.XmlSupport.importMap(XmlSupport.java:388) ~[?:1.8.0]
at java.util.prefs.FileSystemPreferences$6.run(FileSystemPreferences.java:598) ~[?:1.8.0]
at java.util.prefs.FileSystemPreferences$6.run(FileSystemPreferences.java:591) ~[?:1.8.0]
at java.security.AccessController.doPrivileged(AccessController.java:734) ~[?:1.8.0]
at java.util.prefs.FileSystemPreferences.loadCache(FileSystemPreferences.java:590) ~[?:1.8.0]
at java.util.prefs.FileSystemPreferences.initCacheIfNecessary(FileSystemPreferences.java:573) ~[?:1.8.0]
at java.util.prefs.FileSystemPreferences.getSpi(FileSystemPreferences.java:550) ~[?:1.8.0]
at java.util.prefs.AbstractPreferences.get(AbstractPreferences.java:298) ~[?:1.8.0]
at com.ibm.crypto.pkcs11impl.provider.IBMPKCS11Impl.<init>(IBMPKCS11Impl.java:464) ~[ibmpkcs11impl.jar:8.0 build_8.0-20200224-2]
at java.lang.J9VMInternals.newInstanceImpl(Native Method) ~[?:2.9 (06-01-2020)]
at java.lang.Class.newInstance(Class.java:1852) ~[?:2.9 (06-01-2020)]
at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:233) ~[?:1.8.0]
at sun.security.jca.ProviderConfig$2.run(ProviderConfig.java:218) ~[?:1.8.0]
at java.security.AccessController.doPrivileged(AccessController.java:678) ~[?:1.8.0]
at sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:218) ~[?:1.8.0]
at sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:199) ~[?:1.8.0]
at sun.security.jca.ProviderList.getProvider(ProviderList.java:245) ~[?:1.8.0]
at sun.security.jca.ProviderList.getIndex(ProviderList.java:275) ~[?:1.8.0]
at sun.security.jca.ProviderList.getProviderConfig(ProviderList.java:259) ~[?:1.8.0]
at sun.security.jca.ProviderList.getProvider(ProviderList.java:265) ~[?:1.8.0]
at java.security.Security.getProvider(Security.java:479) ~[?:1.8.0]
at com.ibm.jsse2.ac.<clinit>(ac.java:159) ~[?:8.0 build_20200327--103]
at com.ibm.jsse2.ag.<init>(ag.java:18) ~[?:8.0 build_20200327--103]
at com.ibm.jsse2.ag.<init>(ag.java:111) ~[?:8.0 build_20200327--103]
at com.ibm.jsse2.av.a(av.java:87) ~[?:8.0 build_20200327--103]
at com.ibm.jsse2.av.<init>(av.java:636) ~[?:8.0 build_20200327--103]
at com.ibm.jsse2.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:2) ~[?:8.0 build_20200327--103]
at com.singularity.ee.util.httpclient.EasySSLProtocolSocketFactory.createLayeredSocket(EasySSLProtocolSocketFactory.java:135) ~[appagent.jar:?]
at com.singularity.ee.util.httpclient.EasySSLProtocolSocketFactory.connectSocket(EasySSLProtocolSocketFactory.java:193) ~[appagent.jar:?]
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.13.jar:4.5.13]
at com.singularity.ee.util.httpclient.SimpleHttpClientWrapper.executeHttpOperation(SimpleHttpClientWrapper.java:302) ~[appagent.jar:?]
at com.singularity.ee.util.httpclient.SimpleHttpClientWrapper.executeHttpOperation(SimpleHttpClientWrapper.java:217) ~[appagent.jar:?]
at com.singularity.ee.rest.RESTRequest.sendRequestTracked(RESTRequest.java:384) ~[appagent.jar:Server Agent #21.11.3.33314 v21.11.3 GA compatible with 4.4.1.0 r1f94344f9fd88fc14fe39a33494b03e4bb555a6d release/21.11.0]
at com.singularity.ee.rest.RESTRequest.sendRequest(RESTRequest.java:337) ~[appagent.jar:Server Agent #21.11.3.33314 v21.11.3 GA compatible with 4.4.1.0 r1f94344f9fd88fc14fe39a33494b03e4bb555a6d release/21.11.0]
at com.singularity.ee.rest.controller.request.AControllerRequest.sendRequest(AControllerRequest.java:130) ~[appagent.jar:Server Agent #21.11.3.33314 v21.11.3 GA compatible with 4.4.1.0 r1f94344f9fd88fc14fe39a33494b03e4bb555a6d release/21.11.0]
at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.sendRequest(ABinaryControllerRequest.java:36) ~[appagent.jar:Server Agent #21.11.3.33314 v21.11.3 GA compatible with 4.4.1.0 r1f94344f9fd88fc14fe39a33494b03e4bb555a6d release/21.11.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel.registerApplicationServer(ConfigurationChannel.java:1436) ~[appagent.jar:Server Agent #21.11.3.33314 v21.11.3 GA compatible with 4.4.1.0 r1f94344f9fd88fc14fe39a33494b03e4bb555a6d release/21.11.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel.access$100(ConfigurationChannel.java:121) ~[appagent.jar:Server Agent #21.11.3.33314 v21.11.3 GA compatible with 4.4.1.0 r1f94344f9fd88fc14fe39a33494b03e4bb555a6d release/21.11.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel$UnregisteredConfigurationState.nextTransition(ConfigurationChannel.java:784) ~[appagent.jar:Server Agent #21.11.3.33314 v21.11.3 GA compatible with 4.4.1.0 r1f94344f9fd88fc14fe39a33494b03e4bb555a6d release/21.11.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel.refreshConfiguration(ConfigurationChannel.java:554) ~[appagent.jar:Server Agent #21.11.3.33314 v21.11.3 GA compatible with 4.4.1.0 r1f94344f9fd88fc14fe39a33494b03e4bb555a6d release/21.11.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.XMLConfigManager$AgentConfigurationRefreshTask.run(XMLConfigManager.java:656) ~[appagent.jar:Server Agent #21.11.3.33314 v21.11.3 GA compatible with 4.4.1.0 r1f94344f9fd88fc14fe39a33494b03e4bb555a6d release/21.11.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.XMLConfigManager.initialize(XMLConfigManager.java:332) ~[appagent.jar:Server Agent #21.11.3.33314 v21.11.3 GA compatible with 4.4.1.0 r1f94344f9fd88fc14fe39a33494b03e4bb555a6d release/21.11.0]
at com.singularity.ee.agent.appagent.kernel.AgentKernel.start(AgentKernel.java:166) ~[appagent.jar:Server Agent #21.11.3.33314 v21.11.3 GA compatible with 4.4.1.0 r1f94344f9fd88fc14fe39a33494b03e4bb555a6d release/21.11.0]
at com.singularity.ee.agent.appagent.kernel.JavaAgent.initialize(JavaAgent.java:451) ~[appagent.jar:Server Agent #21.11.3.33314 v21.11.3 GA compatible with 4.4.1.0 r1f94344f9fd88fc14fe39a33494b03e4bb555a6d release/21.11.0]
at com.singularity.ee.agent.appagent.kernel.JavaAgent.initialize(JavaAgent.java:346) ~[appagent.jar:Server Agent #21.11.3.33314 v21.11.3 GA compatible with 4.4.1.0 r1f94344f9fd88fc14fe39a33494b03e4bb555a6d release/21.11.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90) ~[?:1.8.0]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) ~[?:1.8.0]
at java.lang.reflect.Method.invoke(Method.java:508) ~[?:1.8.0]
at com.singularity.ee.agent.appagent.AgentEntryPoint$1.run(AgentEntryPoint.java:655) ~[?:Server IBM Agent #21.11.3.33314 v21.11.3 GA compatible with 4.4.1.0 r1f94344f9fd88fc14fe39a33494b03e4bb555a6d release/21.11.0]

Due to this error the agent could not register sucessfully at the controller. Does anyone have an idea how to solve this problem. Is this problem maybe related to the security provider com.ibm.crypto.pkcs11impl.provider.IBMPKCS11Impl which is listed in the file java.security of the jdk on this server? On the other servers which I could sucessfully register at the controller this security provider is not listed in the java.security file. But maybe the cause for the problem is not related to this security provider. 

[iamryan]

Hi @Hans.Mueller,

I found this existing AppD Docs page. Please take a look and let me know if it helps out at all.

https://docs.appdynamics.com/22.2/en/application-monitoring/install-app-server-agents/java-agent/ins...

[arredondolopezh]

I am getting same error and this is happening when migrating onprem to SaaS. please provide next steps

[iamryan]

Hi @Hector A.Arredondo Lopez,

Please see if this Community Knowledge Base Article helps out: https://community.appdynamics.com/t5/Knowledge-Base/How-do-I-resolve-SAAS-java-security-cert/ta-p/19...