Splunk AppDynamics

Agent unable to register with SAAS/On-prem controller

Vinay_Kumar1
New Member

We are trying to deploy Java Agent in our WildFly 10.1.0  Using Java Agent Version [Server Agent #21.3.0.32281 v21.3.0] we are having SAAS contoller.

[AD Thread Pool-Global1] 09 Apr 2021 09:29:59,670 INFO ConfigurationChannel - Sending Registration request with: Application Name [SFA], Tier Name [SEVLSUPP], Node Name [guvwdapsfa01], Host Name [GUVWDAPSFA01] Node Unique Local ID [guvwdapsfa01], Version [Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
[AD Thread-Metric Reporter0] 09 Apr 2021 09:30:00,000 WARN MetricHandler - Metric Reporter Queue full. Dropping metrics.
[AD Thread Pool-Global1] 09 Apr 2021 09:30:00,427 WARN ConfigurationChannel - ResponseReadException creating Response Wrapper [AppAgentConfigurationBinaryRequest], : com.singularity.ee.rest.ResponseReadException: java.lang.UnsupportedOperationException: Deserialization not allowed for class com.singularity.ee.controller.api.constants.AgentType
[AD Thread Pool-Global1] 09 Apr 2021 09:30:00,427 WARN ConfigurationChannel - Unable to get AppAgentConfigurationResponse from controller
[AD Thread Pool-Global0] 09 Apr 2021 09:30:04,185 ERROR NetVizAgentRequest - Fatal transport error while connecting to URL [http://127.0.0.1:3892/api/agentinfo?timestamp=0&agentType=APP_AGENT&agentVersion=1.3.0]: org.apache.http.conn.ConnectTimeoutException: Connect to 127.0.0.1:3892 [/127.0.0.1] failed: connect timed out
[AD Thread Pool-Global1] 09 Apr 2021 09:30:46,921 WARN EventGenerationService - The retention queue is at full capacity [5]. Dropping events for timeslice [Fri Apr 09 09:25:00 IST 2021] to accomodate events for timeslice [Fri Apr 09 09:30:00 IST 2021]
[AD Thread-Metric Reporter1] 09 Apr 2021 09:31:00,000 WARN MetricHandler - Metric Reporter Queue full. Dropping metrics.
[AD Thread Pool-Global1] 09 Apr 2021 09:31:00,474 INFO ConfigurationChannel - Container id retrieval enabled: true
[AD Thread Pool-Global1] 09 Apr 2021 09:31:00,475 INFO ConfigurationChannel - No Container ID found in /proc/self/cgroup : does not exist or is not readable
[AD Thread Pool-Global1] 09 Apr 2021 09:31:00,475 INFO ConfigurationChannel - Agent node meta-info thus far: ProcessID;12164;appdynamics.ip.addresses;192.168.76.60;appdynamicsHostName;GUVWDAPSFA01
[AD Thread Pool-Global1] 09 Apr 2021 09:31:00,475 INFO ConfigurationChannel - Detected node meta info: [Name:ProcessID, Value:12164, Name:appdynamics.ip.addresses, Value:192.168.76.60, Name:appdynamicsHostName, Value:GUVWDAPSFA01, Name:supportsDevMode, Value:true]
[AD Thread Pool-Global1] 09 Apr 2021 09:31:00,475 INFO ConfigurationChannel - Sending Registration request with: Application Name [SFA], Tier Name [SEVLSUPP], Node Name [guvwdapsfa01], Host Name [GUVWDAPSFA01] Node Unique Local ID [guvwdapsfa01], Version [Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
[AD Thread Pool-Global1] 09 Apr 2021 09:31:04,779 ERROR NetVizAgentRequest - Fatal transport error while connecting to URL [http://127.0.0.1:3892/api/agentinfo?timestamp=0&agentType=APP_AGENT&agentVersion=1.3.0]: org.apache.http.conn.ConnectTimeoutException: Connect to 127.0.0.1:3892 [/127.0.0.1] failed: connect timed out
[AD Thread-Metric Reporter1] 09 Apr 2021 09:32:00,001 WARN MetricHandler - Metric Reporter Queue full. Dropping metrics.
[AD Thread Pool-Global0] 09 Apr 2021 09:32:01,280 INFO ConfigurationChannel - Container id retrieval enabled: true
[AD Thread Pool-Global0] 09 Apr 2021 09:32:01,280 INFO ConfigurationChannel - No Container ID found in /proc/self/cgroup : does not exist or is not readable
[AD Thread Pool-Global0] 09 Apr 2021 09:32:01,280 INFO ConfigurationChannel - Agent node meta-info thus far: ProcessID;12164;appdynamics.ip.addresses;192.168.76.60;appdynamicsHostName;GUVWDAPSFA01
[AD Thread Pool-Global0] 09 Apr 2021 09:32:01,280 INFO ConfigurationChannel - Detected node meta info: [Name:ProcessID, Value:12164, Name:appdynamics.ip.addresses, Value:192.168.76.60, Name:appdynamicsHostName, Value:GUVWDAPSFA01, Name:supportsDevMode, Value:true]
[AD Thread Pool-Global0] 09 Apr 2021 09:32:01,280 INFO ConfigurationChannel - Sending Registration request with: Application Name [SFA], Tier Name [SEVLSUPP], Node Name [guvwdapsfa01], Host Name [GUVWDAPSFA01] Node Unique Local ID [guvwdapsfa01], Version [Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
[AD Thread Pool-Global1] 09 Apr 2021 09:32:46,921 WARN EventGenerationService - The retention queue is at full capacity [5]. Dropping events for timeslice [Fri Apr 09 09:27:00 IST 2021] to accomodate events for timeslice [Fri Apr 09 09:32:00 IST 2021]
[AD Thread-Metric Reporter1] 09 Apr 2021 09:33:00,000 WARN MetricHandler - Metric Reporter Queue full. Dropping metrics.
[AD Thread Pool-Global1] 09 Apr 2021 09:33:02,081 INFO ConfigurationChannel - Container id retrieval enabled: true
[AD Thread Pool-Global1] 09 Apr 2021 09:33:02,082 INFO ConfigurationChannel - No Container ID found in /proc/self/cgroup : does not exist or is not readable
[AD Thread Pool-Global1] 09 Apr 2021 09:33:02,082 INFO ConfigurationChannel - Agent node meta-info thus far: ProcessID;12164;appdynamics.ip.addresses;192.168.76.60;appdynamicsHostName;GUVWDAPSFA01
[AD Thread Pool-Global1] 09 Apr 2021 09:33:02,082 INFO ConfigurationChannel - Detected node meta info: [Name:ProcessID, Value:12164, Name:appdynamics.ip.addresses, Value:192.168.76.60, Name:appdynamicsHostName, Value:GUVWDAPSFA01, Name:supportsDevMode, Value:true]
[AD Thread Pool-Global1] 09 Apr 2021 09:33:02,082 INFO ConfigurationChannel - Sending Registration request with: Application Name [SFA], Tier Name [SEVLSUPP], Node Name [guvwdapsfa01], Host Name [GUVWDAPSFA01] Node Unique Local ID [guvwdapsfa01], Version [Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]

Labels (3)
Tags (1)
0 Karma

Vinay_Kumar1
New Member

Can someone please assist here?

Tags (1)
0 Karma

Mark_Byrne
Path Finder

Hi Vinay,

The first thing I would check is connectivity from your host to the AppD SaaS controller - is there a proxy server between them for example?

The second thing would be to look for any alerts in the controller indicating a license error

Mark

0 Karma

Vinay_Kumar1
New Member

Appreciate @Mark.Byrne your response. 

We have multiple machines on the same network. All work fine with other machines.
License is also consumed when we check the controller.
We can telnet to controller as well.

Below message is printed all time in my agent logs

AD Agent init] 08 Apr 2021 16:56:34,592 WARN ConfigurationChannel - Unable to get AppAgentConfigurationResponse from controller
[AD Agent init] 08 Apr 2021 16:56:34,594 INFO XMLConfigManager - Agent will poll for new configuration from controller every [60] seconds.

Today I will install cURL and try to hit my controller and let you know the results.
curl -v https://ABCDE.saas.appdynamics.com:443 or ping

I will also check firewall and proxy issue.

0 Karma

Morelz
Builder

Hi Vinay

Can you share the full agent log if possible?

Ciao

0 Karma

Vinay_Kumar1
New Member

[AD Thread Pool-Global0] 09 Apr 2021 17:00:10,439 INFO ConfigurationChannel - Sending Registration request with: Application Name [SFA], Tier Name [SEVLSUPP], Node Name [guvwdapsfa01], Host Name [GUVWDAPSFA01] Node Unique Local ID [guvwdapsfa01], Version [Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
[AD Thread Pool-Global1] 09 Apr 2021 17:00:10,447 DEBUG SnapshotCountTracker - nForcedSnapshotsStartedThisMin: 0
[AD Thread Pool-Global1] 09 Apr 2021 17:00:10,447 DEBUG SnapshotCountTracker - No limit hit, drop!
[AD Thread Pool-Global0] 09 Apr 2021 17:00:11,362 DEBUG ConfigurationChannel - In AppAgent Config Data error in reading data from servlet response reading byte[], size :141370, error java.lang.UnsupportedOperationException: Deserialization not allowed for class com.singularity.ee.rest.controller.request.NodeProperties
[AD Thread Pool-Global0] 09 Apr 2021 17:00:11,362 DEBUG ConfigurationChannel - com.singularity.ee.rest.ResponseReadException: java.lang.UnsupportedOperationException: Deserialization not allowed for class com.singularity.ee.rest.controller.request.NodeProperties
com.singularity.ee.rest.ResponseReadException: java.lang.UnsupportedOperationException: Deserialization not allowed for class com.singularity.ee.rest.controller.request.NodeProperties
at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.<init>(ABinaryControllerRequest.java:62) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.rest.controller.request.AppAgentConfigurationBinaryRequest.<init>(AppAgentConfigurationBinaryRequest.java:129) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_181]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[?:1.8.0_181]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:1.8.0_181]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:1.8.0_181]
at com.singularity.ee.agent.appagent.kernel.controller.JavaAgentRESTRequestFactory.getResponseObj(JavaAgentRESTRequestFactory.java:203) ~[appagent.jar:?]
at com.singularity.ee.agent.appagent.kernel.controller.JavaAgentRESTRequestFactory.getAppAgentConfigurationResponse(JavaAgentRESTRequestFactory.java:137) ~[appagent.jar:?]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel.registerApplicationServer(ConfigurationChannel.java:1439) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel.access$100(ConfigurationChannel.java:120) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel$UnregisteredConfigurationState.nextTransition(ConfigurationChannel.java:783) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel.refreshConfiguration(ConfigurationChannel.java:553) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.XMLConfigManager$AgentConfigurationRefreshTask.run(XMLConfigManager.java:643) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:122) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_181]
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRunAndReset(ADFutureTask.java:335) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.runAndReset(ADFutureTask.java:152) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$101(ADScheduledThreadPoolExecutor.java:119) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.runPeriodic(ADScheduledThreadPoolExecutor.java:206) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:236) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:694) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:726) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_181]
Caused by: java.lang.UnsupportedOperationException: Deserialization not allowed for class com.singularity.ee.rest.controller.request.NodeProperties
at org.kantega.notsoserial.DefaultNotSoSerial.preventDeserialization(DefaultNotSoSerial.java:240) ~[?:?]
at org.kantega.notsoserial.DefaultNotSoSerial.onBeforeResolveClass(DefaultNotSoSerial.java:231) ~[?:?]
at org.kantega.notsoserial.transformers.ObjectInputStreamClassVisitor.onBeforeResolveClass(ObjectInputStreamClassVisitor.java:49) ~[?:?]
at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source) ~[?:1.8.0_181]
at java.io.ObjectInputStream.readClassDesc(Unknown Source) ~[?:1.8.0_181]
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) ~[?:1.8.0_181]
at java.io.ObjectInputStream.readObject0(Unknown Source) ~[?:1.8.0_181]
at java.io.ObjectInputStream.readObject(Unknown Source) ~[?:1.8.0_181]
at java.util.HashMap.readObject(Unknown Source) ~[?:1.8.0_181]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_181]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_181]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_181]
at java.io.ObjectStreamClass.invokeReadObject(Unknown Source) ~[?:1.8.0_181]
at java.io.ObjectInputStream.readSerialData(Unknown Source) ~[?:1.8.0_181]
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) ~[?:1.8.0_181]
at java.io.ObjectInputStream.readObject0(Unknown Source) ~[?:1.8.0_181]
at java.io.ObjectInputStream.readObject(Unknown Source) ~[?:1.8.0_181]
at com.singularity.ee.util.serialize.JavaSerializer.unmarshalObject(JavaSerializer.java:40) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.constructPaylodFromInputTracked(ABinaryControllerRequest.java:132) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.constructPaylodFromInput(ABinaryControllerRequest.java:124) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.<init>(ABinaryControllerRequest.java:57) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
... 22 more

[AD Thread Pool-Global0] 09 Apr 2021 17:01:11,364 DEBUG SensitiveDataScrubber - Filtering data with key = [javax.net.ssl.trustStorePassword], scrubbed value = [*****]
[AD Thread Pool-Global0] 09 Apr 2021 17:01:11,364 DEBUG SensitiveDataScrubber - Filtering data with key = [javax.net.ssl.keyStoreType], scrubbed value = [*****]
[AD Thread Pool-Global0] 09 Apr 2021 17:01:11,364 DEBUG SensitiveDataScrubber - Filtering data with key = [javax.net.ssl.keyStorePassword], scrubbed value = [*****]
[AD Thread Pool-Global0] 09 Apr 2021 17:01:11,364 DEBUG SensitiveDataScrubber - Filtering data with key = [javax.net.ssl.keyStore], scrubbed value = [*****]
[AD Thread Pool-Global0] 09 Apr 2021 17:01:11,365 DEBUG CloudInstanceIdGetter - Failed to read instance id for appdynamicsAzureVmId
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Connection pool shut down
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerGet(ADFutureTask.java:257) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.get(ADFutureTask.java:101) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledFutureImpl.get(AgentScheduledFutureImpl.java:39) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.cloud.CloudInstanceIdGetter.getCloudInstanceId(CloudInstanceIdGetter.java:76) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.cloud.CloudInstanceIdGetter.getCloudInstanceIdSafe(CloudInstanceIdGetter.java:59) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.ConfigurationChannelRequestDecorator.appendCloudInstanceIdIfEnabled(ConfigurationChannelRequestDecorator.java:191) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.ConfigurationChannelRequestDecorator.setRequestForInitialAgentRegistrationAttempt(ConfigurationChannelRequestDecorator.java:156) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel.registerApplicationServer(ConfigurationChannel.java:1420) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel.access$100(ConfigurationChannel.java:120) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel$UnregisteredConfigurationState.nextTransition(ConfigurationChannel.java:783) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel.refreshConfiguration(ConfigurationChannel.java:553) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.XMLConfigManager$AgentConfigurationRefreshTask.run(XMLConfigManager.java:643) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:122) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_181]
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRunAndReset(ADFutureTask.java:335) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.runAndReset(ADFutureTask.java:152) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$101(ADScheduledThreadPoolExecutor.java:119) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.runPeriodic(ADScheduledThreadPoolExecutor.java:206) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:236) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:694) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:726) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_181]
Caused by: java.lang.IllegalStateException: Connection pool shut down
at org.apache.http.util.Asserts.check(Asserts.java:34) ~[httpcore-4.4.13.jar:4.4.13]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:269) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[httpclient-4.5.13.jar:4.5.13]
at com.singularity.ee.agent.cloud.HttpInstanceIdClient.getInstanceId(HttpInstanceIdClient.java:44) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.cloud.AzureInstanceIdResolver.getInstanceId(AzureInstanceIdResolver.java:45) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.cloud.CloudInstanceIdGetter$InstanceIdResolverCallable.call(CloudInstanceIdGetter.java:98) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.cloud.CloudInstanceIdGetter$InstanceIdResolverCallable.call(CloudInstanceIdGetter.java:88) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$1.call(AgentScheduledExecutorServiceImpl.java:60) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRun(ADFutureTask.java:320) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.run(ADFutureTask.java:140) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$301(ADScheduledThreadPoolExecutor.java:119) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:238) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
... 3 more
[AD Thread Pool-Global0] 09 Apr 2021 17:01:11,365 DEBUG CloudInstanceIdGetter - Failed to read instance id for appdynamicsGcpInstanceId
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Connection pool shut down
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerGet(ADFutureTask.java:257) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.get(ADFutureTask.java:101) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledFutureImpl.get(AgentScheduledFutureImpl.java:39) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.cloud.CloudInstanceIdGetter.getCloudInstanceId(CloudInstanceIdGetter.java:76) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.cloud.CloudInstanceIdGetter.getCloudInstanceIdSafe(CloudInstanceIdGetter.java:59) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.ConfigurationChannelRequestDecorator.appendCloudInstanceIdIfEnabled(ConfigurationChannelRequestDecorator.java:191) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.ConfigurationChannelRequestDecorator.setRequestForInitialAgentRegistrationAttempt(ConfigurationChannelRequestDecorator.java:156) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel.registerApplicationServer(ConfigurationChannel.java:1420) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel.access$100(ConfigurationChannel.java:120) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel$UnregisteredConfigurationState.nextTransition(ConfigurationChannel.java:783) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel.refreshConfiguration(ConfigurationChannel.java:553) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.XMLConfigManager$AgentConfigurationRefreshTask.run(XMLConfigManager.java:643) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:122) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_181]
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRunAndReset(ADFutureTask.java:335) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.runAndReset(ADFutureTask.java:152) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$101(ADScheduledThreadPoolExecutor.java:119) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.runPeriodic(ADScheduledThreadPoolExecutor.java:206) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:236) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:694) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:726) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_181]
Caused by: java.lang.IllegalStateException: Connection pool shut down
at org.apache.http.util.Asserts.check(Asserts.java:34) ~[httpcore-4.4.13.jar:4.4.13]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:269) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[httpclient-4.5.13.jar:4.5.13]
at com.singularity.ee.agent.cloud.HttpInstanceIdClient.getInstanceId(HttpInstanceIdClient.java:44) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.cloud.GcpInstanceIdResolver.getInstanceId(GcpInstanceIdResolver.java:44) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.cloud.CloudInstanceIdGetter$InstanceIdResolverCallable.call(CloudInstanceIdGetter.java:98) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.cloud.CloudInstanceIdGetter$InstanceIdResolverCallable.call(CloudInstanceIdGetter.java:88) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$1.call(AgentScheduledExecutorServiceImpl.java:60) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRun(ADFutureTask.java:320) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.run(ADFutureTask.java:140) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$301(ADScheduledThreadPoolExecutor.java:119) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:238) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
... 3 more
[AD Thread Pool-Global0] 09 Apr 2021 17:01:11,365 DEBUG CloudInstanceIdGetter - Failed to read instance id for appdynamicsAwsInstanceId
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Connection pool shut down
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerGet(ADFutureTask.java:257) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.get(ADFutureTask.java:101) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledFutureImpl.get(AgentScheduledFutureImpl.java:39) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.cloud.CloudInstanceIdGetter.getCloudInstanceId(CloudInstanceIdGetter.java:76) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.cloud.CloudInstanceIdGetter.getCloudInstanceIdSafe(CloudInstanceIdGetter.java:59) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.ConfigurationChannelRequestDecorator.appendCloudInstanceIdIfEnabled(ConfigurationChannelRequestDecorator.java:191) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.ConfigurationChannelRequestDecorator.setRequestForInitialAgentRegistrationAttempt(ConfigurationChannelRequestDecorator.java:156) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel.registerApplicationServer(ConfigurationChannel.java:1420) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel.access$100(ConfigurationChannel.java:120) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel$UnregisteredConfigurationState.nextTransition(ConfigurationChannel.java:783) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel.refreshConfiguration(ConfigurationChannel.java:553) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.XMLConfigManager$AgentConfigurationRefreshTask.run(XMLConfigManager.java:643) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:122) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_181]
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRunAndReset(ADFutureTask.java:335) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.runAndReset(ADFutureTask.java:152) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$101(ADScheduledThreadPoolExecutor.java:119) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.runPeriodic(ADScheduledThreadPoolExecutor.java:206) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:236) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:694) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:726) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_181]
Caused by: java.lang.IllegalStateException: Connection pool shut down
at org.apache.http.util.Asserts.check(Asserts.java:34) ~[httpcore-4.4.13.jar:4.4.13]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:269) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[httpclient-4.5.13.jar:4.5.13]
at com.singularity.ee.agent.cloud.HttpInstanceIdClient.getInstanceId(HttpInstanceIdClient.java:44) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.cloud.AwsInstanceIdResolver.getInstanceId(AwsInstanceIdResolver.java:39) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.cloud.CloudInstanceIdGetter$InstanceIdResolverCallable.call(CloudInstanceIdGetter.java:98) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.cloud.CloudInstanceIdGetter$InstanceIdResolverCallable.call(CloudInstanceIdGetter.java:88) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$1.call(AgentScheduledExecutorServiceImpl.java:60) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRun(ADFutureTask.java:320) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.run(ADFutureTask.java:140) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$301(ADScheduledThreadPoolExecutor.java:119) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:238) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
... 3 more
[AD Thread Pool-Global0] 09 Apr 2021 17:01:11,365 INFO ConfigurationChannel - Container id retrieval enabled: true
[AD Thread Pool-Global0] 09 Apr 2021 17:01:11,366 INFO ConfigurationChannel - No Container ID found in /proc/self/cgroup : does not exist or is not readable
[AD Thread Pool-Global0] 09 Apr 2021 17:01:11,366 INFO ConfigurationChannel - Agent node meta-info thus far: ProcessID;11348;appdynamics.ip.addresses;192.168.76.60;appdynamicsHostName;GUVWDAPSFA01
[AD Thread Pool-Global0] 09 Apr 2021 17:01:11,366 INFO ConfigurationChannel - Detected node meta info: [Name:ProcessID, Value:11348, Name:appdynamics.ip.addresses, Value:192.168.76.60, Name:appdynamicsHostName, Value:GUVWDAPSFA01, Name:supportsDevMode, Value:true]
[AD Thread Pool-Global0] 09 Apr 2021 17:01:11,366 INFO ConfigurationChannel - Sending Registration request with: Application Name [SFA], Tier Name [SEVLSUPP], Node Name [guvwdapsfa01], Host Name [GUVWDAPSFA01] Node Unique Local ID [guvwdapsfa01], Version [Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
[AD Thread Pool-Global1] 09 Apr 2021 17:01:11,911 DEBUG SegmentManager - Future task map size 0 for 0 active transactions:
[AD Thread Pool-Global1] 09 Apr 2021 17:01:12,144 DEBUG InstrumentationSdkPluginManager - Polling instrumentation sdk plugin directory C:\Titan\SFA\AppServerAgent-21.3.0.32281\ver21.3.0.32281\sdk-plugins
[AD Thread Pool-Global1] 09 Apr 2021 17:01:12,258 DEBUG SizeReporter - Not reporting metrics MemoryMonitor[false], objcts[0]
[AD Thread Pool-Global0] 09 Apr 2021 17:01:12,313 DEBUG ConfigurationChannel - In AppAgent Config Data error in reading data from servlet response reading byte[], size :141370, error java.lang.UnsupportedOperationException: Deserialization not allowed for class com.singularity.ee.rest.controller.request.NodeProperties
[AD Thread Pool-Global0] 09 Apr 2021 17:01:12,313 DEBUG ConfigurationChannel - com.singularity.ee.rest.ResponseReadException: java.lang.UnsupportedOperationException: Deserialization not allowed for class com.singularity.ee.rest.controller.request.NodeProperties
com.singularity.ee.rest.ResponseReadException: java.lang.UnsupportedOperationException: Deserialization not allowed for class com.singularity.ee.rest.controller.request.NodeProperties
at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.<init>(ABinaryControllerRequest.java:62) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.rest.controller.request.AppAgentConfigurationBinaryRequest.<init>(AppAgentConfigurationBinaryRequest.java:129) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_181]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[?:1.8.0_181]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:1.8.0_181]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:1.8.0_181]
at com.singularity.ee.agent.appagent.kernel.controller.JavaAgentRESTRequestFactory.getResponseObj(JavaAgentRESTRequestFactory.java:203) ~[appagent.jar:?]
at com.singularity.ee.agent.appagent.kernel.controller.JavaAgentRESTRequestFactory.getAppAgentConfigurationResponse(JavaAgentRESTRequestFactory.java:137) ~[appagent.jar:?]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel.registerApplicationServer(ConfigurationChannel.java:1439) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel.access$100(ConfigurationChannel.java:120) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel$UnregisteredConfigurationState.nextTransition(ConfigurationChannel.java:783) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.ConfigurationChannel.refreshConfiguration(ConfigurationChannel.java:553) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.agent.appagent.kernel.config.xml.XMLConfigManager$AgentConfigurationRefreshTask.run(XMLConfigManager.java:643) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:122) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_181]
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRunAndReset(ADFutureTask.java:335) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.runAndReset(ADFutureTask.java:152) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$101(ADScheduledThreadPoolExecutor.java:119) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.runPeriodic(ADScheduledThreadPoolExecutor.java:206) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:236) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:694) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:726) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_181]
Caused by: java.lang.UnsupportedOperationException: Deserialization not allowed for class com.singularity.ee.rest.controller.request.NodeProperties
at org.kantega.notsoserial.DefaultNotSoSerial.preventDeserialization(DefaultNotSoSerial.java:240) ~[?:?]
at org.kantega.notsoserial.DefaultNotSoSerial.onBeforeResolveClass(DefaultNotSoSerial.java:231) ~[?:?]
at org.kantega.notsoserial.transformers.ObjectInputStreamClassVisitor.onBeforeResolveClass(ObjectInputStreamClassVisitor.java:49) ~[?:?]
at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source) ~[?:1.8.0_181]
at java.io.ObjectInputStream.readClassDesc(Unknown Source) ~[?:1.8.0_181]
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) ~[?:1.8.0_181]
at java.io.ObjectInputStream.readObject0(Unknown Source) ~[?:1.8.0_181]
at java.io.ObjectInputStream.readObject(Unknown Source) ~[?:1.8.0_181]
at java.util.HashMap.readObject(Unknown Source) ~[?:1.8.0_181]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_181]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_181]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_181]
at java.io.ObjectStreamClass.invokeReadObject(Unknown Source) ~[?:1.8.0_181]
at java.io.ObjectInputStream.readSerialData(Unknown Source) ~[?:1.8.0_181]
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) ~[?:1.8.0_181]
at java.io.ObjectInputStream.readObject0(Unknown Source) ~[?:1.8.0_181]
at java.io.ObjectInputStream.readObject(Unknown Source) ~[?:1.8.0_181]
at com.singularity.ee.util.serialize.JavaSerializer.unmarshalObject(JavaSerializer.java:40) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.constructPaylodFromInputTracked(ABinaryControllerRequest.java:132) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.constructPaylodFromInput(ABinaryControllerRequest.java:124) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.<init>(ABinaryControllerRequest.java:57) ~[appagent.jar:Server Agent #21.3.0.32281 v21.3.0 GA compatible with 4.4.1.0 rcd7d7317f698cefb9003377c1af0ff3ef81ee922 release/21.3.0]
... 22 more

0 Karma

iamryan
Community Manager
Community Manager

Hi @Vinay.Kumar,

Your reply was caught as spam. It looked like two replies was caught as spam, one was longer, so I removed the reply that was not as long. 

@Vinay.Kumar I'd also like to note that be careful when sharing logs on the Community. This is a public site that can be seen by anyone. Be sure to scrub any sensitive information from logs before posting them on Community. If you wish to be more cautious, share logs via Private Message with other members. 

0 Karma

Oliver_Zakaria
Explorer

Any update here ??
Because I got the same error in on premise controller. Here the logs :
[AD Thread Pool-Global0] 19 May 2021 10:32:58,309 WARN NetVizConfigurationChannel - NetViz: Number of communication failures with netviz agent exceeded maximum allowed [3]. Disabling config requests.
[AD Thread Pool-Global0] 19 May 2021 10:33:58,302 WARN ConfigurationChannel - ResponseReadException creating Response Wrapper [AppAgentConfigurationBinaryRequest], : com.singularity.ee.rest.ResponseReadException: Error in controller in processing binary request AppAgent Config Data - Application component node id:218 is not associated to machine id:206
[AD Thread Pool-Global0] 19 May 2021 10:33:58,310 ERROR NetVizAgentRequest - Fatal transport error while connecting to URL [http://127.0.0.1:3892/api/agentinfo?timestamp=0&agentType=APP_AGENT&agentVersion=0.11.1]: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:3892 [/127.0.0.1] failed: Connection refused (Connection refused)
[AD Thread Pool-Global0] 19 May 2021 10:34:09,015 INFO ControllerTimeSkewHandler - Controller Time Skew Handler Run Aborted - Skew Check is Disabled
[AD Thread Pool-Global1] 19 May 2021 10:34:58,307 WARN ConfigurationChannel - ResponseReadException creating Response Wrapper [AppAgentConfigurationBinaryRequest], : com.singularity.ee.rest.ResponseReadException: Error in controller in processing binary request AppAgent Config Data - Application component node id:218 is not associated to machine id:206
[AD Thread Pool-Global1] 19 May 2021 10:34:58,315 ERROR NetVizAgentRequest - Fatal transport error while connecting to URL [http://127.0.0.1:3892/api/agentinfo?timestamp=0&agentType=APP_AGENT&agentVersion=0.11.1]: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:3892 [/127.0.0.1] failed: Connection refused (Connection refused)
[AD Thread Pool-Global0] 19 May 2021 10:35:58,312 WARN ConfigurationChannel - ResponseReadException creating Response Wrapper [AppAgentConfigurationBinaryRequest], : com.singularity.ee.rest.ResponseReadException: Error in controller in processing binary request AppAgent Config Data - Application component node id:218 is not associated to machine id:206

0 Karma

Morelz
Builder

Hi Oliver

The error you are seeing is mainly related to duplicate node names.

Please check that no other node is already registered with the Controller with the same node name.

You can update the node name to a unique value

Ciao

Get Updates on the Splunk Community!

Splunk Observability Synthetic Monitoring - Resolved Incident on Detector Alerts

We’ve discovered a bug that affected the auto-clear of Synthetic Detectors in the Splunk Synthetic Monitoring ...

Video | Tom’s Smartness Journey Continues

Remember Splunk Community member Tom Kopchak? If you caught the first episode of our Smartness interview ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud? Learn how unique features like ...