Health rules  are based on one or more of an application's performance metrics and let you specify the parameters for what is considered “normal” for your application. When the health rule is violated, an event is raised that triggers a health policy.  Health policies can also be triggered by other types of events (e.g. errors). In sum, health rules work on metrics, policies work on events, and the evaluation of health rules triggers events on which policies take action. For a diagram of how this functions, see Alert and Respond. Below are the most common reasons and steps to take if your health policies aren’t triggering. Contents Confirm that the health rule violation is being triggered Confirm that the event that triggers the policy is occurring What to check if the event exists What to check if the event doesn't exist 1. Confirm that the health rule violation is being triggered If your policies are triggered by health rule violation events and your health rules aren’t configured properly, violations may not be fired. Review your health rule configuration - Ensure you followed the workflows outlined in Configure Health Rules when creating your health rule. Check your Health Rule Evaluation configuration - This is the “Use the last <x> minutes of data when evaluating the health rule” field and the default is 30 minutes. This is the window of time in which the violation is detected, rather than a range over which the violation must persist. See the "Health Rule Evaluation Window" section in this document for additional information. Check your Wait Time after Violation configuration - This value is used to avoid alert storms after a violation has occurred. When a violation is detected, the health rule is not re-evaluated until after this wait time has expired. If the erroneous conditions correct themselves before that window of time expires, you won’t receive any additional alerts. See the "Health Rule Wait Time After Violation" section in this document for details. 2. Confirm that the event that triggers the policy is occurring Policies can be fired by health rule violation events or events that are automatically raised under certain conditions, such as errors and slow transactions. Events can also be manually registered or created programmatically using the REST API. For a list of events, see Events Reference. If you are on an older version of AppDynamics, refer to the events list for your specific version. If you do not see the expected event in the Controller UI, evaluate the following: Slow transactions - Check the defined threshold and the baseline. Not allowing sufficient time for the baseline to be calculated or a lack of load on the application can affect the ability to define the baseline. If there is no baseline, we can’t monitor of the deviations from the baseline. Errors - Check if the agent logs contain the expected error. If the error is found but the event didn’t trigger, contact AppDynamics Support. Code Problems - Resource Pool Limit Reached is a predefined event that can trigger a policy. Policies for this event are not configured by default, so you would need to create one. The event is triggered by 80% usage of EJB-related resource pools (e.g. connection pools and thread pools), but that threshold can be configured using the jms-metric-threshold-percentage node property. Changing the value requires a restart of the agent JVM to apply this change. Type: Integer Default: 80% Platform: Java Create discrete policies for health rule events - Rather than create one all-encompassing policy, create more specific ones so you can better understand the scope of the policy configuration. Before creating a policy, read: Configuring Policies For more information, see: Policy Triggers 3. If the event exists, check the following: Evaluated object - When the health rule is defined, you select the objects that you want monitored. If you create a policy based on the health rule, you can also select the objects that the policy applies to. If there is a mismatch between the health rule definition and the policy definition, the policy will not work as expected. If the event was not on the expected object, edit the health rule and/or policy configuration. Check the expected policy action - You can assign the actions that are taken when a policy is triggered: The user running the script has executable permissions The dependent files or directory exist The Machine Agent is up and running or reachable by the Controller If the expected action is to send an email, check if the email server is working. If the expected action is to run a remediation Machine Agent script, check the following: If the expected action is to take a thread dump, check if the agent node is running. If the expected action is to run a Custom Action, validate your configuration. 4. If the event doesn’t exist, check the following: Evaluation time - Metrics that are being evaluated against a baseline need time and load to build up the baseline. If the timeframe or the load is insufficient, evaluation against the baseline can't occur. Confirm metrics exist - When metrics are discovered, they are registered with the Controller. If something happens to the entities that identify the source of specific metrics (e.g., the application, tier, or node), the health rule evaluation will fail and there won’t be a rule violation to trigger the associated policy. Symptoms (besides no event or "alert") include log entries in Controller logs similar to the following: [#|SEVERE|glassfish3.1.2|com.appdynamics.RULES.PROCESSING|_ThreadID=75;_ThreadName=Thread-6;|An Error occured while evaluating Policy com.singularity.ee.controller.api.exceptions.ObjectNotFoundException: Metric not found: <metric_id> at com.singularity.ee.controller.beans.manage.policies.PolicyProcessorBean$SingleThreadedRuleProcessorBean.evaluateLeafCondition(PolicyProcessorBean.java:1895) Run the following query on the Controller database: mysql> use controller; mysql> select id, name, application_id from metric where id in (<metric_IDs from controller log errors>); . The query should return empty set (no results) if the metrics actually don't exist. If a tier, node or specific metric used in a health rule has been deleted, the behavior may be uncertain. The metric must exist or the health rule can’t operate properly. From the command line, change to the Controller's bin directory and use the appropriate script to log in to the Controller database: For Windows: controller.bat login-db For Linux: sh controller.sh login-db If an incorrect metric was used, select the correct one and re-apply load on the application.   ... View more

Errors In some cases, database corruption can result in java errors, deadlocks, sporadic Controller shutdowns, and Controller inaccessibility. Below is a list of errors that may indicate database corruption. If you encounter one of them or other issues that you think could indicate corruption, please contact AppDynamics Support so we can assist you. Error 1: InnoDB: cannot calculate statistics for table controller/<table_or_partition_name> InnoDB: because the .ibd file is missing. For help, please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting.html 180302 17:40:45 [ERROR] MySQL is trying to open a table handle but the .ibd file for table controller/<table_or_partition_name> does not exist.​ Have you deleted the .ibd file from the database directory under the MySQL datadir, or have you used DISCARD TABLESPACE? See http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting.html how you can resolve the problem. Error 2: InnoDB: Error: page <page_number> log sequence number <log_sequence_number> InnoDB: is in the future! Current system log sequence number <log_sequence_number>. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: for more information. Error 3: [ERROR] mysqld.exe: Incorrect key file for table 'sql.MYI'; try to repair it [ERROR] Got an error from unknown thread, ..\..\..\mysqlcom-pro-5.5.40\storage\myisam\mi_write.c:<line_number> InnoDB: Error: tablespace id is <tablespace_id> in the data dictionary InnoDB: but in file <file_name>.ibd it is <tablespace_id>! InnoDB: Assertion failure in thread <thread_id> in file fil0fil.c line <line_number> InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. Error 4: InnoDB: Assertion failure in thread <thread_id> in file mtr0log.c line <line_number> InnoDB: Failing assertion: DATA_ROLL_PTR_LEN == dict_index_get_nth_col(ind, DATA_ROLL_PTR - 1 + n_uniq)->len InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. Error 5: InnoDB: Error: trying to access page number <page_number> in space <space_id>, InnoDB: space name <ibd_file_name>, InnoDB: which is outside the tablespace bounds. InnoDB: Byte offset <offset>, len <length>, i/o type <type_number>. InnoDB: If you get this error at mysqld startup, please check that InnoDB: your my.cnf matches the ibdata files that you have in the InnoDB: MySQL server. Prepare to contact Support It is important that you do not attempt to fix database corruption without hands-on assistance from our Support team. If you notice one of the errors above, please contact Support and prepare the following information:   Logs files: Please share all the platform log files: Platform Log Files If the Controller version is older than 4.4, provide Controller log files: Controller Log Files If you have an HA installation, provide HA log files from both primary and secondary servers: HA Log Files Is the Controller server on a physical machine or a virtual machine? If on a virtual machine, is the RAM fully reserved? If so, please share the screenshot of memory reservation: Fully Reserved RAM What is the Controller profile and does the hardware meet the system requirements: System Requirements If your Controller is on Linux, provide the /var/log/messages file for the date of the database crash and the event viewer logs for the MySQL service. Information about series of events like a Controller migration, Controller restart or a server reboot that happened before the database corruption. ... View more

To make sure you have all of the information you need to understand license consumption, rules, errors, and usage, we've indexed several articles below that will guide you through how to resolve common issues and answer frequently asked questions. Table of Contents Licensing Functionality  Troubleshooting  Viewing License Consumption and Usage History  Licensing Functionality These articles will provide you with a basic understanding of how licenses are consumed and how to best use license rules to strategically allocate units: How does AppDynamics license consumption work? How is Java Agent license consumption calculated? How is .NET Agent license consumption calculated? What are license rules and how do they work? What happens if I delete a license rule? What are some best practices for license management? Troubleshooting If you encounter any notifications or errors related to licensing, please review the troubleshooting resources linked below: Why am I seeing license limit notifications? How do I resolve "Agent License Request Denied" license limit notifications? Viewing License Consumption and Usage History The resources below will guide you through how to view license consumption from different perspectives to help you better understand your usage: How do I view license usage and history on the Subscription page? How do I check my license usage at the account level? How do I find license usage by applications for each controller? ... View more

In a large MySQL database with numerous tables and partitions, the  information_schema.partitions table is built serially and is cached once built. This is so the App Server can be restarted quickly if the database is not also restarted. For this reason, it is not recommended to restart the database unless absolutely necessary. To track how long the Controller takes to start, run the following command from the computer terminal during startup repeatedly at two-minute intervals until the Controller UI is accessible: echo "select time, info from information_schema.processlist where command = 'Query' and user != 'root';" | <controller_home>/bin/controller.sh login-db  If the results include the  information_schema.partitions , the Controller is building the list of partitions and the Controller UI would not be accessible yet. If frequent database restarts are necessary, the amount of time to start can be improved by adding the innodb_stats_sample_pages=1 property to the end of the <controller_home>/db/db.cnf file (after "# End of standard db.cnf" ) if it does not exist already and restarting the Controller. This property is added by default for High Availability Controller Deployments. Example: # End of standard db.cnf # added to speed up startup innodb_stats_sample_pages=1 ... View more

When the Controller is set up for High Availability, users see a "Controller disabled on this host" message on the secondary Controller when a partial replication is run and full replication with the -f flag is not yet complete. When a replication runs without the -f flag, it copies the MySQL data and index files from the primary server to the secondary while App Server (Glassfish) runs on the primary server. There may be tables updated by App Server thus changing the underlying data files, but these files may have already been copied over to the secondary server. This is an incremental sync. Running a replication with the -f flag stops the Controller on the primary server, ensuring that no files will change, then compares the files between the primary and secondary servers and only copies those that are different. This is final sync. More incremental syncs create less downtime when running a replication with the -f flag, as there are fewer files to copy in the direct sync. To ensure MySQL does not start on secondary server until the final sync is complete, replicate script deletes the  <controller_home>/bin/controller.sh  file on the secondary server until the replication is finished and hence the message "Controller disabled on this host" so that controller cannot be started on secondary. Once the final replication completes,<controller_home>/bin/controller.sh is automatically copied from primary to secondary server and MySQL on secondary server is started. ... View more

Users installing their Controller on a Windows machine may see the following error message on the installer screen: Please resolve the errors and retry the operation. A Microsoft Windows hotfix is required but is not detected on this system. Please install the hotfix from http://support.microsoft.com/kb/2549760 and retry the operation. This hotfix ensures that the windows registry modifications made by this installer to extend the default service timeouts work as expected. If you are unable to install the hotfix, send the file C:\Users<Username?\AppData\Local\Temp\hotfix_status_XXXXXXX.txt and the installation log in your temp directory to AppDynamics support for assistance. The error message appears when hotfix #2549760 has not been installed on the Windows server.  Solution Bypass the hotfix validation during Controller installation: Launch a command prompt as an Administrator on your Windows machine. Run the Controller installer with a -Vskip_validation=true flag to bypass the hotfix check. Example: <controller_installer_name>.exe -Vskip_validation=true Note: There is no loss of functionality without the hotfix, but be aware that until the hotfix is installed, users must stop all Controller services (including app and database) and ensure they have completely shut down before rebooting the Controller. This prevents database corruption which may be irreversible. Installing the hotfix after successful Controller installation: Shutdown all Controller related services (app and database). Install the hotfix by running the following commands in a command prompt as an Administrator: reg add HKLM\SYSTEM\CurrentControlSet\Control /v WaitToKillServiceTimeout /t REG_SZ /d 600000 /f   Increase service stop timeouts reg add HKLM\SYSTEM\CurrentControlSet\Control /v ServicesPipeTimeout /t REG_SZ /d 1200000 /f Increases service startup timeouts Start the controller services. ... View more

How do I resolve "Network Communication Error" in the Controller UI? Table of Contents Syptoms Diagnosis Solution Symptoms The following "Network Communication Error" appears in the Controller UI: Diagnosis Error message in <controller_home>/logs/server.log : [INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.l ogging|_ThreadID=58;_ThreadName=Thread-5;|[BlazeDS] 05/24/2017 16:07:50.527 [ERROR] [Endpoint.General] Unexpected error encountered in Message Broker servlet flex.messaging.security.SecurityException: Secure endpoint '/messagebroker/amfsecure' must be contacted via a secure protocol. at flex.messaging.endpoints.AbstractEndpoint.validateRequestProtocol(AbstractEndpoint.java:1368) at flex.messaging.endpoints.AbstractEndpoint.service(AbstractEndpoint.java:935) at flex.messaging.endpoints.BaseHTTPEndpoint.service(BaseHTTPEndpoint.java:312) at flex.messaging.MessageBrokerServlet.service(MessageBrokerServlet.java:373) at javax.servlet.http.HttpServlet.service(HttpServlet.java:770) at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1550) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:343 ) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:217) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:89) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130) at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter[INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.l ogging|_ThreadID=58;_ThreadName=Thread-5;|[BlazeDS] [ERROR] [Endpoint.General] Unexpected error encountered in Message Broker servlet flex.messaging.security.SecurityException: Secure endpoint '/messagebroker/amfsecure' must be contacted via a secure protocol. at flex.messaging.endpoints.AbstractEndpoint.validateRequestProtocol(AbstractEndpoint.java:1368) at flex.messaging.endpoints.AbstractEndpoint.service(AbstractEndpoint.java:935) at flex.messaging.endpoints.BaseHTTPEndpoint.service(BaseHTTPEndpoint.java:312) at flex.messaging.MessageBrokerServlet.service(MessageBrokerServlet.java:373) at javax.servlet.http.HttpServlet.service(HttpServlet.java:770) at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1550) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:343 ) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:217) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:89) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130) at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256 ) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:217) at com.singularity.ee.controller.servlet.CsrfFilter.doFilter(CsrfFilter.java:84) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256 ) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:217) at com.singularity.ee.controller.servlet.RequestOriginMarkingFilter.doFilter(RequestOriginMarkingFi lter.java:26) Solution This issue occurs when an SSL-terminating proxy is redirecting requests to the Controller and the Controller is not configured to handle these requests. To resolve this issue, follow the steps int the documentation under "Configure the Controller for SSL Termination at the Proxy." ... View more