We recently had a short metric gaps in the controller UI (SaaS Controller) for several apps and different agents (DB, App and Machine). The log files of all the different agents all have a common ...
See more...
We recently had a short metric gaps in the controller UI (SaaS Controller) for several apps and different agents (DB, App and Machine). The log files of all the different agents all have a common theme: "Connection back off limitation in effect" "Fatal transport error while connecting to URL" also comes up sometimes as a similar error logged by agents. I did a quick search online and this seems to be an AppD agent specific log file entry. The AppD community also had about 12 entries going back to 2017, all with no clear solution to this error message. (Summary below) Docs site search returns nothing. I opened an AppD support case and will see what they say, but it is frustrating to see that this is a common thing reported by different agents without a clear cause for it documented anywhere. I wonder why something like this is logged the way it is which makes me think its something to do with a limitation on the Controller side of things, when all other community posts and agent logs make it look like it is not Controller related. Examples of our recent issue: * I tried to redact the important bits DB Agent v23.2.2 [Entity-Registration-Scheduler-19] 31 Oct 2023 10:50:25,932 WARN EntityRegistrar - Fail to register [DBSession] entities: java.lang.RuntimeException: Connection back off limitation in effect: /controller/instance/***/registerServerSatelliteEntity at com.singularity.ee.agent.dbagent.task.reporter.EntityRegistrar.registerEntities(EntityRegistrar.java:276) ~[db-agent.jar:Database Agent v23.2.0.0 GA compatible with 4.5.2.0 Build Date 2023-02-22] Other DB Agent v23.2.2 [<**DB Collector Name***>-Transient-Event-Scheduler-2] 31 Oct 2023 10:51:22,737 WARN SystemAgentTransientEventChannel - Error sending event data to controller: Connection back off limitation in effect: /controller/instance/***/transient-channel Different DB agent v23.8.8 [<**DB Collector Name***>-Scheduler-3] 31 Oct 2023 10:51:52,288 INFO ADBCollector - Collected one-minute data for *** [Entity-Registration-Scheduler-2] 31 Oct 2023 10:51:52,850 WARN EntityRegistrar - Fail to register [Query] entities: java.lang.RuntimeException: Connection back off limitation in effect: /controller/instance/3945944/registerSQLQuery SIM (Machine)Agent **ServerName**==> [AD Thread-Metric Reporter0] 31 Oct 2023 10:51:56,554 ERROR ManagedMonitorDelegate - Error sending metrics - will requeue for later transmission com.singularity.ee.agent.commonservices.metricgeneration.metrics.MetricSendException: Connection back off limitation in effect: /controller/instance/***/metrics SIM Agent v22x ***Hostname***==> [AD Thread-Metric Reporter0] 31 Oct 2023 10:51:48,204 ERROR ManagedMonitorDelegate - Fatal transport error while connecting to URL [/controller/instance/***/metrics]: org.apache.http.conn.ConnectTimeoutException: Connect to ***:443 [***/***, ***, ***] failed: connect timed out ***Hostname***==> [AD Thread-Metric Reporter0] 31 Oct 2023 10:51:48,204 WARN ManagedMonitorDelegate - Error sending metric data to controller:Fatal transport error while connecting to URL [/controller/instance/***/metrics] ***Hostname***==> [AD Thread-Metric Reporter0] 31 Oct 2023 10:51:48,204 ERROR ManagedMonitorDelegate - Error sending metrics - will requeue for later transmission com.singularity.ee.agent.commonservices.metricgeneration.metrics.MetricSendException: Fatal transport error while connecting to URL [/controller/instance/***/metrics] Summary of other AppD community posts with a similar error from agent log files: 2017 Community post https://community.appdynamics.com/t5/NET-Agent-Installation/Azure-Cloud-Service-No-load-detected-App-agent-status-0/td-p/26538 No solutions in ticket/unresolved 2017 Community post no 2 https://community.appdynamics.com/t5/Dynamic-Languages-Node-JS-Python/Could-not-connect-to-the-controller-invalid-response-from/td-p/28680 Python agent issues Mentions proxy setup for outbound requests from agent server, but no clear answer other than bringing the node online on controller, whatever that means 2017 Community post no3 https://community.appdynamics.com/t5/NET-Agent-Installation/Failed-to-add-web-app-to-AppDynamics/td-p/23699 No confirmed solution, but last posts suggests using non ssl settings which is not a great solution if that is the fix 2017 Community post no4 https://community.appdynamics.com/t5/NET-Agent-Installation/net-Agent-registering-issue/td-p/29595 Proxy setting highlighted but no ultimate solution 2018 Community post https://community.appdynamics.com/t5/NET-Agent-Installation/BT-requests-and-survival/td-p/29629 Answers do not address the "Connection back off limitation in effect" issue 2018 Community post no2 https://community.appdynamics.com/t5/NET-Agent-Installation/After-NET-Agent-upgrade-to-4-3-7-1-we-are-not-seeing-load-for/td-p/34528 Issue shown in one log file extract but not addressed 2018 Community post no3 https://community.appdynamics.com/t5/Controller-SaaS-On-Premises/Unable-to-connect-to-the-controller/td-p/30857 No final solution 2018 Community post no4 https://community.appdynamics.com/t5/NET-Agent-Installation/Need-help-on-installation-of-agent/td-p/34673 Post never had a resolution 2019 Community post https://community.appdynamics.com/t5/NET-Agent-Installation/no-metrics-in-controller-after-net-agent-installation-in-linux/td-p/37848 Possible issue with AppDynamicsConfig.json No clear answer/solution 2019 Community post no2 https://community.appdynamics.com/t5/NET-Agent-Installation/Net-core-agent-Linux-is-not-connecting-to-the-saas-controller/td-p/37867 No solution 2021 Community post https://community.appdynamics.com/t5/Knowledge-Base/How-do-I-install-the-NET-Core-Microservices-Agent-for-Windows/ta-p/33191 Answers do not address the "Connection back off limitation in effect" issue 2023 Community post https://community.appdynamics.com/t5/Controller-SaaS-On-Premises/Could-not-connect-to-the-controller-invalid-response-from/m-p/50571#M3319 Suggests ignoring or disabling the errors Here is to hoping there is a solution or better answer to this issue.