What should I consider when resolving agent metric registration limits? In this article... Why do I see this error and what can I do about it? AGENT_METRIC_BLACKLIST_REG_LIMIT_REACHED  How can we find the custom metric with an incorrect configuration?  How can we determine the correlation between metric size and CPU utilization? Can other system utilization also be affected by configuration? Why do I see this error and what can I do about it?  AGENT_METRIC_BLACKLIST_REG_LIMIT_REACHED When an agent tries to register more than 5000 metrics, it will fire the error below to the Controller. The default limit is 5,000 per session and once it has been met, the agent will not register new metrics.  NOTE | A new session begins at the last restart  This error is most often seen when users try to register Custom Metrics with incorrect configurations that consume the metric limit. AGENT_METRIC_BLACKLIST_REG_LIMIT_REACHED To resolve the AGENT_METRIC_BLACKLIST_REG_LIMIT_REACHED error message: First, delete unwanted business transactions, information points, service endpoints, and backends. If all of the metrics are still required, you can increase the agent metric registration limit using the system property below and restarting the agent. -Dappdynamics.agent.maxMetrics=<IntegerNumber> Use caution when changing a metric registration limit. A drastic increase in the agent limit can affect the Controller. It is highly recommended that you check your current CPU and memory utilization from the application side before increasing the metric registration limit. Restart the agent. NOTE | Identify which metrics are registered, and focus only on relevant metrics. Increasing the limits every time you encounter this error can cause serious performance issues. How can we find the custom metric with an incorrect configuration? There is no single place to look for an incorrect configuration of the custom metric. If you find you’re hitting the backend limit, check the backend configuration.  Similarly, if you’re exceeding limits in SEP, BT, etc., look in those places for how you might adjust or reconfigure them to bring your numbers down. When increasing the maximum metric size, how can we anticipate how large the increase to higher CPU utilization will be? Increasing the maximum metric size can cause high CPU utilization, but only at a very high value. The Controller also has a metric limit of 1M per Controller application, including all its tiers and nodes.  As a rule of thumb, things should work fine if you keep below 1M per Controller. Can utilization of the main Controller also be affected by configuration? If the Controller application metric limit is exceeding limits even after you've increased those limits, then the Controller level can also experience issues as a result. ... View more

If your Application Agent is not reporting data to the Controller, checking the following items on the machine where the Agent is installed: Is the Agent log empty? Are there network connectivity issues? Are there 401 Unauthorized error messages in the Agent logs? Is the ssl-enabled property value set to true? Are the application, tier, and node names correct? Are you using a multi-tenant Controller? Are there license-related errors? 1. Is the Agent log empty? If no data is being reported to the Controller, it’s possible that the user who is starting the application server doesn’t have the necessary permissions on the Agent directory. The user requires read, write and execute permissions. The Agent should also be installed with the same user with which the application process is running. Example: You may use chown -R user:group appdynamics and your user is "jboss" chown -R jboss:jboss appdynamics. This gives the Application Agent user “jboss” the complete ability to navigate the /opt/appdynamics folders and sub folders. If user "jboss" doesn’t have the permission to navigate to root-owned folders within AppDynamics, the Agent wouldn’t be able to write to log files that are required by the application. To resolve this issue, check the permissions in the Agent directory and sub-directories. Example: <AgentDir>/ver4.x.x.x/ 2. Are there network connectivity issues? Use the following command to test connectivity: curl -v https://<controllerHost>:<port>/controller telnet <controllerHost> traceroute <controllerHost>  Check with your Network engineer to resolve connectivity issues between the Agent and Controller. 3. Are there 401 Unauthorized error messages in the Agent logs? This error message usually indicates that the account name or access key are missing or incorrectly configured. 4. Is the ssl-enabled property value set to true? If the Agent is using SSL port 8181 and you see the following error message in the logs, check that the ssl-enabled property has a value of true in the controller-info.xml file or any parameters configured in startup file. [Thread-0] 22 Nov 2017 09:09:43,469 ERROR ConfigurationChannel - Exception: athenetest.saas.appdynamics.com:443 failed to respond org.apache.http.NoHttpResponseException: athenetest.saas.appdynamics.com:443 failed to respond Ensure that SSLEnabled is set to true for HTTPS, and false for HTTP. 5. Are the application, tier, and node names correct? These are mandatory configuration parameters for an Application Agent. Look at Agent logs for errors like this: 6. Are you using a multi-tenant Controller? If yes, check that the account name is accurate. 7. Are there license-related errors? Check the Agent logs for license-related errors and verify if the license count limit has been reached: WARN ConfigurationChannel - ResponseReadException creating Response Wrapper [e], : com.singularity.ee.rest.b: Error in controller in processing binary request AppAgent Config Data - Agent license request denied. If you see a license-related error, follow the steps outlined in Why am I seeing license limit notifications? Additional troubleshooting resources: Can you guide me to resources for agent-related issues? Troubleshooting Agent Issues - FAQs Troubleshooting Agent Issues - Installation and Start-up Why is the Java Agent not reporting to the Controller? Last Updated: 3/27/19 ... View more

 Problem : In Sim dashboard we are seeing only the rpool volume and other volumes are not reporting   This Is By Design:  According to the current implementation for the solaris machine we report only volumes which are "/dev/dsk" apart from rpool volumes.    Please refer to https://docs.oracle.com/cd/E19455-01/805-6331/6j5vgg680/index.html, "Convention for Disks" , We report Disk devices, the naming convention is "/dev/dsk" .    Currently we will not show other volume apart from rpool and /dev/dsk voulmes for solaris   _______________________________________________ Was this helpful? Give it kudos [click on the thumbs up icon at the end of the article, bottom left corner] to let others know too!   Do you have a question or need clarification? [Post Your Comment] Below.    ... View more

How do I solve an issue where my SIM/Machine Agent isn't capturing metrics, or showing them in the Network tab? Problem SIM/Machine Agent is not capturing Network metrics, and we are not seeing metrics in the Network tab   Troubleshooting network metrics First, check to see whether third-party interface multiplexors or any NIC card configuration is being used. If this is the case,  SIM does not currently handle third-party interfaces or NIC card configuration, so the blank page under the Network tab would be expected. If the Network metrics are still not being monitored, try switching to Sigar API. However, this option means we will lose all of the SIM functionality. ... View more

In order to calculate the volumes I/O metrics, a Machine Agent must use the disk file system command or df as well as the cat /proc/diskstats command in Linux. Then the Machine Agent matches the mount point and file system names of both of the command outputs. Example:   If the output of df is: Linux-VirtualBox:~$ df Filesystem 1K-blocks Used Available Use% Mounted on udev 368708 4 368704 1% /dev tmpfs 75884 948 74936 2% /run /dev/sda1 11479264 6694412 4178696 62% / and the output of proc/diskstats is:  Linux-VirtualBox:~$ cat /proc/diskstats 8 0 sda 45542 1694 1800102 44732 173309 171383 3502920 37456 0 43884 82128 8 1 sda1 42589 483 1766554 43576 161862 141753 3175080 33336 0 41848 76856 As  /dev/sda1 is mounted on / (root) , the partition name will be  dev-sda1 and the disk stats for sda1 is mapped to it. So now on the Controller UI, the I/O metric for dev-sda1 will be displayed as the same I/O metric which displays in the SIM dashboard for / (root) volume. In what circumstance will a Machine Agent not report the I/O metric for volumes? If the volumes mounted on volume groups, the Machine Agent will not report the I/O metrics for volume.   Example output of  df : Linux-VirtualBox:~$:/home/adm/cosg# df Filesystem 1048576-blocks Used Available Capacity Mounted on /dev/mapper/vg00-lv_root 1984 1624 258 87% / /dev/mapper/vg00-lv_tmp 496 51 420 11% /tmp /dev/mapper/vg00-lv_home 93 7 82 8% /home ... View more

My application is start-up is taking longer after I added an AppDynamics Agent. Why does this happen and how do I resolve it? Delays in application start-up time after adding an Agent can be due to the Agent requesting configuration details from the JVM. By default, the Agent reserves 60 seconds to get a response from the Controller and start the instrumentation process. In some cases, it takes longer to get a response which in turn causes a delay. Change the following system property to 3 seconds, and the instrumentation request will be issued in parallel with application startup. Changing the parameter will have no impact on the application. How do I resolve these application delays? You can resolve the delay by changing the timeout system property from the default 60 seconds to 3 seconds. The instrumentation request will be issued in parallel with application startup. This parameter change will not have any impact on the application.  While running the agent, add the following system property to the Agent configuration: -Dappagent.start.timeout=3000 Example:   java -Dappagent.start.timeout=3000 -javaagent:<AgentInstallDir>/verx.x.x/javaagent.jar​ ... View more

Issue While running the docker-compose up command, the Docker container for the Machine Agent is not running. The following permission error is shown. Could not start up the machine agent due to: Failed to get a response from /info using a GET request. The error encountered is: XXXX:XXXXX:XXXX: Permission denied (socket: /run/docker.sock) ​ This issue occurs on a default installation when a user other than the root user, or a user outside of the Docker group, is trying to access the socket. Solution First, check to make sure the mount is properly configured. Then, if you have eliminated configuration as the cause, run the following command in the machine terminal.   sudo usermod -a -G docker $username Finally, log out of the account completely, and log back in.  Related Links: Solving Docker permission denied while trying to connect to the Docker daemon socket Video: Docker Visibility explained  If you have any issues viewing the above video please email education@appdynamics.com. ... View more

If an AppDynamics Agent fails to instrument JDBC driver for Vertica database, Vertica backend calls will not be visible.  To diagnose this issue, use the following API calls to verify if Agent is able to apply the interceptors. com.vertica.jdbc.SPreparedStatement, com.vertica.jdbc.VerticaPreparedStatement com.vertica.jdbc.SCallableStatement,com.vertica.jdbc.SStatement com.vertica.jdbc.VerticaStatement,com.vertica.jdbc.VerticaConnection com.vertica.jdbc.SConnection, com.vertica.jdbc.SConnectionHandle To resolve the issue, add the following classes for each of the four node-level properties. jdbc-prepared-statements=com.vertica.jdbc.SPreparedStatement,com.vertica.jdbc.VerticaPreparedStatement jdbc-callable-statements = com.vertica.jdbc.SCallableStatement jdbc-statements= com.vertica.jdbc.SStatement,com.vertica.jdbc.VerticaStatement jdbc-connections = com.vertica.jdbc.VerticaConnection,com.vertica.jdbc.SConnection,com.vertica.jdbc.SConnectionHandle Follow the documentation to add a registered node property. ... View more

Question Why is the memory usage reported by Machine Agent higher than reported by monitoring tools such as "htop" and "top"? Answer The Machine Agent memory metrics include the total memory used by the processes as well as memory used by the kernel. For example, the metrics include memory used by buffering and disk cache in a Linux environment. Whereas, the "htop" or "top" commands show only the memory used by the processes. ... View more

Question Why is the Java Agent reporting a large number of com.singularity.ReflectionError messages? Is there a negative impact of these errors on the Agent? Answer The Java Agent uses a Reflection API at runtime to find class information. During this process, if the Agent encounters a framework which doesn't comply with J2EE standards, or if there is no support for said framework, the error messages appear in the Agent logs. These errors often occur when the user has a framework which is not compatible with the Agent (such as different versions), or the user defined a rule which includes an incorrect "getter chain" (info point, custom exit, POJO, etc). The error messages do not impact the Agent's performance or functionality. ... View more

Problem The top 10 processes are not appearing on the SIM Dashboard, and the process list is not showing all the processes in the Controller UI. In the Controller logs, the following error message appears, indicating that the maximum limit of SIM processes that the Machine agent can send to the Controller has been reached. limit, sim.processes.count.maxPerMachine, is set to 1000 processes. This limit has been reached and will be reset after the next purging. Could not create 136 processes for machine 7|#]     Solution There are two options to solve this problem.  1. Increase the value of the sim.processes.count.maxPerMachine property. Navigate to the Controller Adminstator page (http://<hostname>:<port>/controller/admin.jsp) Log into the system account with the root user password.  Click on Controller Settings. Search for sim.processes.count.maxPerMachine count and increase the value from 1000 to the number of processes you want to monitor. Note: Increasing this setting can affect the resource consumption of your deployment. Before you increase this setting, verify that your application environment and Controller can handle the increased resource requirements. 2. Control the number of processes being monitored. Users who want to monitor only certain processes can open the <MachineAgentInstallDir>/extensions/ServerMonitoring/ServerMonitoring.yml file and enter the monitored process names in the processSelectorRegex field. ... View more

Symptoms While running the Machine Agent, you may see the following error in the Machine Agent logs: ERROR RawCollectorUtil - Could not collect raw data com.fasterxml.jackson. databind.JsonMappingException: No content to map due to end-of-input at [Source: ; line: 1, column: 1] at com.fasterxml.jackson. databind.JsonMappingException. from(JsonMappingException. java:148) at com.fasterxml.jackson. databind.ObjectMapper._ initForReading(ObjectMapper. java:3747) at com.fasterxml.jackson. databind.ObjectMapper._ readMapAndClose(ObjectMapper. java:3687) at com.fasterxml.jackson. databind.ObjectMapper. readValue(ObjectMapper.java: 2714) at com.appdynamics.sim.agent. extensions.servers.model. RawCollectorUtil.runCollector( RawCollectorUtil.java:101) at com.appdynamics.sim.agent. extensions.servers.model. RawCollectorUtil.runCollector( RawCollectorUtil.java:67) at com.appdynamics.sim.agent. extensions.servers.model. newlinux.NewLinuxRawCollector. collectRawData( NewLinuxRawCollector.java:62) at com.appdynamics.sim.agent. extensions.servers.model. newlinux.NewLinuxRawCollector. collectRawData( NewLinuxRawCollector.java:36) at com.appdynamics.sim.agent. extensions.servers.model. Server.collectAndReport( Server.java:43) at com.appdynamics.sim.agent. extensions.servers. ServersMonitor.run( ServersMonitor.java:90) at java.util.concurrent. Executors$RunnableAdapter. call(Executors.java:511) at java.util.concurrent. FutureTask.runAndReset( FutureTask.java:308) at java.util.concurrent. ScheduledThreadPoolExecutor$ ScheduledFutureTask.access$ 301( ScheduledThreadPoolExecutor. java:180) at java.util.concurrent. ScheduledThreadPoolExecutor$ ScheduledFutureTask.run( ScheduledThreadPoolExecutor. java:294) at java.util.concurrent. ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1142) at java.util.concurrent. ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread. java:745) ERROR RawCollectorUtil - The standard out from the collector script was:  WARN RawCollectorUtil - The error log from the collector script was: ERROR: missing ip When this occurs, the Machine Agent is not able to collect machine-level metrics. Diagnosis This error occurs when the IP command is not in the PATH for the Machine Agent user. Without the correctly configured PATH, it is impossible for the Machine Agent to use SIM to collect the network metadata. To verify the root cause, run the following commands in the shell prompt on the server that runs the Machine Agent. shell> ip a shell > which ip shell> echo $PATH The following results confirm the root cause: bash: ip: command not found which ip /usr/bin/which: no ip in no ip present is path Solution 1) Stop the Machine Agent. 2) Set IP as an environment variable for the user who is running the Machine Agent.  "ip" utility is usually in /sbin folder.  Check if /sbin/ip -V returns ip version and if it returns then ip is in /sbin . Add /sbin to PATH environment variable Reference: ip(8) - Linux man page 3) Start the Machine Agent. ... View more

Symptoms Use this split rule if you have an arraylist <String>, and you need to fetch the value of pincode in the data collector so that anyone can use that data collector in analytics to retrieve results. java.util.ArrayList<String> customerList = new java.util.ArrayList<String>(); customerList.add("name=foo"); customerList.add("pincode=67869"); customerList.add("country=null"); customerList.add("state=LA"); Diagnosis You create a Java split rule and it's formatted as follows: getcustomerList.get(1).split("=")[1];   However, you try this split rule in the data collector and it doesn't work.   Solution In your Java split rule, format it as follows: getcustomerList.get(int/1).split(=).[1] See also: Using Getter Chains - Java ... View more

When you will get Error and Error/Min Metrics for Information Point Whenever there is unhandled exception and you are creating an information point on that Class/Method, it will collect Error and Error/Min metrics for that information point. The code snippet will look like as mentioned below:       public void getResult () throws NumberFormatException{            String name = "abc";            Integer.parseInt((name));          } In the above case, only the information point will show you Error and Error/Min metrics and you will not get any Error and Error/Min metric for a BT if you created a POJO Custom BT on same Class/Method. When you will not get Error and Error/Min Metrics for Information Point Whenever you are catching those exception using logger, you will get Error and Error/Min metrics for the BT, not for the information point. The code snippet will look like this:     public void getResult() {     try {          String name = "abc";          Integer.parseInt((name));         } catch (NumberFormatException e) {           logger.log(Level.SEVERE, e.getMessage());         }       } In above case, you will get Error and Error/Min metrics for the BT if you created either a POJO Custom BT or Default BT, but if you created an information point of the same Class/Method, it will not show you Error and Error/Min metrisc. ... View more