All TKB Articles in Learn Splunk

Learn Splunk

All TKB Articles in Learn Splunk

Symptoms You encounter an issue with the POST to /eumcollector/beacons. JS error is "Request header field X-TS-AJAX-Request is not allowed by Access-Control-Allow-Headers". Diagnosis Re... See more...
Symptoms You encounter an issue with the POST to /eumcollector/beacons. JS error is "Request header field X-TS-AJAX-Request is not allowed by Access-Control-Allow-Headers". Diagnosis Research revealed that Access-Control-Allow-Headers is a CORS property.   For this scenario, there were two app environments: APP1 and APP2. EUM data was being seen in APP1 but not in APP2. Verified that the adrum.js and adrum-ext.[ver].js match except for app key. Both apps were hitting the same load balancer, F5. Only APP2 has SSO enabled via WebSphere + Active Directory. Solution Turns out that this was an F5 configuration error, even though "everything's the same for both environments" with regard to F5.   Customer was running the Application Security Manager module on F5. It's supposed to be transparent, not do anything. The F5 team disabled the ASM view on APP2 because it was disabled on APP1. This fixed the CORS problem affecting the call to /eumcollector/beacons.
You can assign multiple MIDCs to the same class and method or multiple information points to the same class and method. However, you cannot assign more than one type of interceptor to the same class ... See more...
You can assign multiple MIDCs to the same class and method or multiple information points to the same class and method. However, you cannot assign more than one type of interceptor to the same class and method in a .NET application. For example, defining an information point or a method invocation data collector (MIDC) on the same class and method that is already instrumented as an entry or exit point .
JDBC connection pool metrics are not configured out-of-the-box for GlassFish. To configure them, uncomment the JDBC connection pool section and provide the relevant information in the following f... See more...
JDBC connection pool metrics are not configured out-of-the-box for GlassFish. To configure them, uncomment the JDBC connection pool section and provide the relevant information in the following file: <agent_home>/ver<version_number>/conf/jmx/servers/glassfish-v2-jmx-config.xml Uncomment the following section and follow the instructions provided in the file. <!-- The following config can be uncommented to monitor glassfish JDBC connection pool. Please set the name of the connection pool (not the datasource name) and enable monitoring for the JDBC Pools on glassfish admin console. --> <!-- <metric mbean-name-pattern="com.sun.appserv:type=jdbc-connection-pool,category=monitor,name=<set the name of pool>,*" category="JDBC Connection Pools"> <attribute-counter-mappings> <attribute-counter-mapping> <attribute-name>numconnused-current</attribute-name> <counter-name>Connections In Use</counter-name> <counter-type>average</counter-type> <time-rollup-type>average</time-rollup-type> <cluster-rollup-type>individual</cluster-rollup-type> </attribute-counter-mapping> <attribute-counter-mapping> <attribute-name>numconnused-highwatermark</attribute-name> <counter-name>Max Connections Used</counter-name> <counter-type>observation</counter-type> <time-rollup-type>average</time-rollup-type> <cluster-rollup-type>individual</cluster-rollup-type> </attribute-counter-mapping> <attribute-counter-mapping> <attribute-name>numpotentialconnleak-count</attribute-name> <counter-name>Potential Leaks</counter-name> <counter-type>observation</counter-type> <time-rollup-type>average</time-rollup-type> <cluster-rollup-type>individual</cluster-rollup-type> </attribute-counter-mapping> <attribute-counter-mapping> <attribute-name>averageconnwaittime-count</attribute-name> <counter-name>Avg Wait Time Millis</counter-name> <counter-type>observation</counter-type> <time-rollup-type>average</time-rollup-type> <cluster-rollup-type>individual</cluster-rollup-type> </attribute-counter-mapping> <attribute-counter-mapping> <attribute-name>waitqueuelength-count</attribute-name> <counter-name>Current Wait Queue Length</counter-name> <counter-type>observation</counter-type> <time-rollup-type>average</time-rollup-type> <cluster-rollup-type>individual</cluster-rollup-type> </attribute-counter-mapping> </attribute-counter-mappings> </metric>
In some situations, JMX metrics from GlassFish are not reported. Also, some metrics may not be enabled by default. Try this solution: Confirm that JMX monitoring is enabled in the GlassFish serv... See more...
In some situations, JMX metrics from GlassFish are not reported. Also, some metrics may not be enabled by default. Try this solution: Confirm that JMX monitoring is enabled in the GlassFish server.  Navigate to the  <agent_home>/ver<version_number>/conf/jmx/ directory.  Copy the attached XML into an mbean-servers.xml file in that directory.  You should see a new JMX node in the metrics tree.
In certain situations, you may encounter the following exception in the agent.log file for the Java Agent deployed on the WebSphere Application Server (WAS). [AD Thread-Transient Event Channel Pol... See more...
In certain situations, you may encounter the following exception in the agent.log file for the Java Agent deployed on the WebSphere Application Server (WAS). [AD Thread-Transient Event Channel Poller0] 17 Aug 2011 08:14:08,031 ERROR JMXTransientOperationsHandler - Error trying to lookup clz - java.lang.ClassNotFoundException: com.ibm.ws.security.core.SecurityContext   To resolve this issue: From the WAS administration console, navigate to the JVM settings for the server of interest: Application servers -> <server> -> Process Definition -> Java Virtual Machine. Remove the following setting from the generic JVM settings: -Djavax.management.builder.initial = -Dcom.sun.management.jmxremote
Java Web Start software allows you to download and run Java applications from the web and is included in the Java Runtime Environment (JRE) since release of Java 5.0. Java Web Start applications are ... See more...
Java Web Start software allows you to download and run Java applications from the web and is included in the Java Runtime Environment (JRE) since release of Java 5.0. Java Web Start applications are launched by using the Java Network Launch Protocol (JNLP). Environment  Java client application launched via JNLP JVM: Oracle 1.7.0_71 Java(TM) SE Runtime Environment (build 1.7.0_71-b14) OS: Windows 7 64-bit    Solution To attach the Java Agent, use one of the following approaches: Approach 1   javaws -J-javaagent:<agent-install- dir>\javaagent.jar <jnlp file>   Approach 2 set JAVA_HOME=C:\Program Files\Java\jre7 set JAVA_TOOL_OPTIONS=-javaagent:< agent-install-dir>\javaagent. jar<JAVA_HOME>\bin\javaws.exe <jnlp file> Security Settings If the agent is not behaving as expected due to security related issues, then you might need to tweak Java security settings as follows: Find the <JAVA_HOME>\lib\security\ javaws.policy file . Edit the file with these settings:  grant codeBase "file:${jnlpx.home}/javaws.jar" { permission java.security.AllPermission; }; grant { permission java.security.AllPermission; };
Updated 7/23/18 Issue I am receiving the following notification on the Controller. CONTROLLER_METRIC_DATA_BUFFER_OVERFLOW   Explanation This message indicates that the data buffe... See more...
Updated 7/23/18 Issue I am receiving the following notification on the Controller. CONTROLLER_METRIC_DATA_BUFFER_OVERFLOW   Explanation This message indicates that the data buffer used to store the metrics is full before these can be flushed to the underlying data store. The buffer is an in-memory cache used to store metrics temporarily in the Controller Appserver and is emptied periodically after the metric data is written to the persistence store. If the buffer becomes full before it is emptied, then the metric data buffer overflow error is generated. When this occurs, the new metrics are dropped by the Controller until space is available in the buffer. The buffer can be intermittently or consistently full. The main determining factors that affect this metric flow are the metric ingestion rate and I/O throughput (and latency) of the underlying storage system. The buffers are sized according to the Controller profile but can vary depending upon the environment. If the buffers are consistently full, then it mostly means the incoming metrics rate is high and buffers are not sized accordingly. If the buffers are intermittently full, it means there is a sudden spike in the metrics rate and/or the I/O throughput of storage is not sufficient enough to flush the metrics in a timely manner. It’s usually the latter and is often seen in SAN-based storages. Example: Let's say the metrics data buffer is sized as 300MB and it can hold approximately 1 million metric data points.   1) If each minute metrics rate is <= 1 million/min and the throughput to write metrics to storage is 1+ million/min, then the buffers will not overflow. [Ideal situation] 2) If the incoming metrics rate is 2 million/min, then the buffer will be full every minute because buffers can only hold 1 million at any given time and extra metrics will potentially be dropped. [Buffer not sized correctly] 3) If the incoming metrics rate is ~1 million/min and the disk write throughput is not fast enough, then buffers would still be required to hold data worth more than 1 minute and potentially will get full since they’re not getting flushed to keep up with the incoming rate. [Slow disk write] Solution 1) If it’s determined that the metrics buffer is not sized properly, increasing the buffer size will fix the problem. The approximate calculation we use for metrics buffer size is a 300-400 MB per one million metrics/min metrics rate. This size considers extra space to hold metrics data for 1+ minute(s) worth of data. The buffer uses the Controller's heap memory; therefore, it’s important that the Controller's host has enough RAM (reserved) available that can be allocated to the Controller’s heap if required. If you have an on-prem Controller, log in to the admin.jsp page of the Controller by logging out of the existing account and going to url <host>/controller/admin.jsp . Set the value for the Controller setting = "metrics.buffer.size" to a higher value and restart the Controller server. The buffers are sized at the Controller startup, so any change in buffer requires a Controller Appserver restart. If you have a SaaS Controller, the buffer sizes are usually set appropriately but a sudden spike can lead to buffer overflow. If you notice overflow notifications, contact the AppDynamics Support team. 2) If the metrics buffer size is correctly set, then most likely the underlying cause of buffer overflow is slow disk I/O throughput. Check the Controller profile, sizing and disk I/O requirements outlined here: Controller System Requirements If you need further assistance, contact AppDynamics Support team.
Question A count of HTTP error codes appears on the main dashboard. Is there a way to get a list of the specific error codes, such as 301, 401, 404, 500 and so on? Answer In the Controller UI,... See more...
Question A count of HTTP error codes appears on the main dashboard. Is there a way to get a list of the specific error codes, such as 301, 401, 404, 500 and so on? Answer In the Controller UI, go to Troubleshoot -> Errors, select the Exceptions tab, and type HTTP error into the search box to filter by "HTTP error".
Question How can I debug my controller install? Answer The Controller installer actually keeps a verbose log of what it is doing on every platform. The filename is of the general format i4j_lo... See more...
Question How can I debug my controller install? Answer The Controller installer actually keeps a verbose log of what it is doing on every platform. The filename is of the general format i4j_log_Controller_nnnnnnnnnnnnnnnnnnn.log. On Linux, you can find it in the /tmp directory. On Windows it is in the user's temp folder. (Type %TEMP% into the windows explorer location bar). On Mac, once you start the installer, open up a terminal window and type: lsof | grep Controller You will see a result like this: 86112 /private/var/folders/np/ctwhy04n6mxdbvhlw_yjkq_cpslbk3/T/i4j_log_Controller_3263273107085015708.log You can "tail -f" that filename to track the installer's detailed progress.
Question How can I check the expiration date of the SSL certificate being used by my Controller? Answer 1. Use the following command and change localhost:8181 to match the controller you are q... See more...
Question How can I check the expiration date of the SSL certificate being used by my Controller? Answer 1. Use the following command and change localhost:8181 to match the controller you are querying. echo ^D |openssl s_client -connect localhost:8181 | openssl x509 -noout -enddate 2. You can also use: http://exchange.nagios.org/directory/Plugins/Network-Protocols/HTTP/check_ssl_certificate/details.
I need to instrument an Adobe ColdFusion app — how do I do it? Table of Contents How do I install the Java Agent on Cold Fusion? Tips: ColdFusion with Java Agent Resource Summary H... See more...
I need to instrument an Adobe ColdFusion app — how do I do it? Table of Contents How do I install the Java Agent on Cold Fusion? Tips: ColdFusion with Java Agent Resource Summary How do I install the Java Agent on Cold Fusion? Install the AppDynamics Java Agent on ColdFusion using either the ColdFusion Administrator Console method, or the ColdFusion .jvm config file method, found below: Install the Java Agent using the ColdFusion Administrator Console Start ColdFusion and access the Administrator Console.  Expand SERVER SETTINGS in the left side tree, then click Java and JVM. In the JVM Arguments text box, add the AD agent entry to the -javaagent argument. Click Submit Changes. Install the Java Agent using the ColdFusion jvm.config file  You can also add the -javaagent argument to jvm.args property in jvm.config file located at  <ColdFusion_Install_Dir>\cfusion\bin\jvm.config and restart the ColdFusion.  For example: shell>>cd C:\ColdFusion10\cfusion\bin shell> coldfusion.exe restart Ex: C:\ColdFusion10\cfusion\bin\jvm.config Note: Additional configuration is required for transaction discovery. Refer to AppDynamics Servlet Entry Points documentation Tips: ColdFusion with Java Agent The Java Agent can handle ColdFusion applications that target the JVM. A ColdFusion application, once compiled to Java class files, is not that different from the typical Java app. The agent can identify business transactions, exit points, and so on. CfmServlet is detected as entry point on the JRun Server, out of the box. The Agent also has support for mapping ColdfFusion UDF's (User defined Functions) in callgraphs. Resource Summary AppDynamics  Servlet Entry Points documentation
This topic describes the tier and node life cycle within AppDynamics. Phase 1 - Agent Startup Phase 2 - Registration Phase 3 - Metric Registration Phase 4 - Metric Reporting Phase 5 - Metri... See more...
This topic describes the tier and node life cycle within AppDynamics. Phase 1 - Agent Startup Phase 2 - Registration Phase 3 - Metric Registration Phase 4 - Metric Reporting Phase 5 - Metric Rollup A combination of agent logs and REST logs reveal the life cycle details for the tier and node metrics. Phase 1 - Agent Startup The application name, tier name, and node name are passed to the agent as part of the start-up process. For Java, the following JVM arguments are used: -Dappdynamics.agent.applicationName -Dappdynamics.agent.tierName -Dappdynamics.agent.nodeName Example log snippet [Thread-0] 21 May 2014 21:15:16,041 INFO AgentKernel - JVM Args : ... | -javaagent:/apps/appdynamics/current/javaagent.jar | -Dappdynamics.controller.hostName=appdynamics.<domain>.net | -Dappdynamics.controller.port=8090 | -Dappdynamics.agent.applicationName=CL-Tax-WL-prod | -Dappdynamics.agent.tierName=GoWeb2-prod | -Dappdynamics.agent.nodeName=GoWeb05-02-prod | -Dappdynamics.agent.runtime.dir=/apps/Java/GoWeb/appdynamics ... From these JVM arguments, you can see the application name is CL-Tax-WL-prod, tier name is GoWeb2-prod, and the node name is GoWeb05-02-prod. -Dappdynamics.agent.applicationName=CL-Tax-WL-prod | -Dappdynamics.agent.tierName=GoWeb2-prod | -Dappdynamics.agent.nodeName=GoWeb05-02-prod Phase 2 - Registration The agent sends the registration request to the controller to register the application, tier, and node. [Thread-0] 21 May 2014 21:15:17,220  INFO ConfigurationChannel - Sending Registration request with: Application Name [CL-Tax-WL-prod], Tier Name [GoWeb2-prod], Node Name [GoWeb05-02-prod], Host Name [fada1wap05] Node Unique Local ID [GoWeb05-02-prod], Version [Server Agent v3.6.1.0 GA #2012-12-11_14-59-42 r2346f2ff54e68a0be000f975f9d0ebc828f8b660 42] Controller Provides/Confirms the IDs The component ID is the internal designation for a tier.   [Thread-0] 21 May 2014 21:15:17,982 INFO ConfigurationChannel - Auto agent registration SUCCEEDED! [Thread-0] 21 May 2014 21:15:17,982 INFO ConfigurationChannel - Registration information received Node ID[153] Component ID[57] Application ID [16]   Phase 3 - Metric Registration The metrics are reported from each agent/node, and registered with the controller. Tier level metrics look like this in the logs: BTM|Application Summary|Component:10|Calls per Minute BTM|Application Summary|Component:10|Exit Call:HTTP|To:14|Calls per Minute BTM|Application Summary|Component:7|Exit Call:JDBC|To:{[UNRESOLVED][9]}|Calls per Minute Log entries from the agent logs for metric registration and reporting for these tier-level metrics: Calls per Minute Errors per Minute Stall Count Number of Slow Calls [AD Thread-Metric Reporter0] 21 Mar 2014 11:28:30,708 DEBUG MetricHandler - 0 BTM|Application Summary|Component:16|Calls per Minute 28 UNREGISTERED 612018265 [AD Thread-Metric Reporter0] 21 Mar 2014 11:28:30,707 DEBUG MetricHandler - 0 BTM|Application Summary|Component:16|Errors per Minute 0 UNREGISTERED 277551759 [AD Thread-Metric Reporter0] 21 Mar 2014 11:28:30,708 DEBUG MetricHandler - 0 BTM|Application Summary|Component:16|Stall Count 0 UNREGISTERED 2010399551 [AD Thread-Metric Reporter0] 21 Mar 2014 11:28:30,708 DEBUG MetricHandler - 0 BTM|Application Summary|Component:16|Number of Slow Calls 0 UNREGISTERED 1630292838 Metric IDs applied to the unregistered metrics: [AD Thread-Metric Reporter0] 21 Mar 2014 11:28:40,706 DEBUG MetricHandler - adding unregistered metric BTM|Application Summary|Component:16|Errors per Minute [AD Thread-Metric Reporter0] 21 Mar 2014 11:28:40,707 DEBUG MetricHandler - adding unregistered metric BTM|Application Summary|Component:16|Stall Count [AD Thread-Metric Reporter0] 21 Mar 2014 11:28:40,707 DEBUG MetricHandler - adding unregistered metric BTM|Application Summary|Component:16|Number of Slow Calls [AD Thread-Metric Reporter0] 21 Mar 2014 11:28:40,707 DEBUG MetricHandler - adding unregistered metric BTM|BTs|BT:241|Component:16|Errors per Minute The following logs show the assignment of the metric ID to specific metrics. These logs are found in the REST log files. <metric id="7527" name="BTM\|Application Summary\|Component:16\|Number of Slow Calls"/> <metric id="7499" name="BTM\|BTs\|BT:241\|Component:16\|Errors per Minute"/> <metric id="7526" name="BTM\|Application Summary\|Component:16\|Stall Count"/> <metric id="7530" name="BTM\|Application Summary\|Component:16\|Calls per Minute"/> Phase 4 - Metric Reporting Metrics reported using the metric IDs. These logs are from the REST log files. <metric id='7527', value\[sum=0, count=1, min=0, max=0, current=0\]> <metric id='7499', value\[sum=0, count=1, min=0, max=0, current=0\]> <metric id='7526', value\[sum=0, count=1, min=0, max=0, current=0\]> <metric id='7530', value\[sum=28, count=1, min=28, max=28, current=28\]> Phase 5 - Metric Rollup This stage happens at the controller. The metrics from all nodes are rolled up in the controller according to the values of time-rollup-type and cluster-rollup-type that was specified at the time of metric registration. The metric registration logs can be found in the REST log. They look similar to the following: <node-id>35</node-id> <agent-type>APP_AGENT</agent-type> <metric time-rollup-type="AVERAGE" name="BTM|Application Summary|Component:16|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|Application Summary|Component:16|Infrastructure Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="SUM" name="BTM|Application Summary|Number of Slow Calls" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="SUM" name="BTM|Application Summary|Stall Count" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="SUM" name="BTM|Application Summary|Component:16|Stall Count" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="SUM" name="BTM|Application Summary|Component:16|Number of Slow Calls" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="SUM" name="BTM|Application Summary|Number of Slow End to End Messages" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="COLLECTIVE"/>
Understanding the AppDynamics error detection life cycle can help you to troubleshoot missing errors and related error metrics. Use agent log files to search for evidence of each phase of error detec... See more...
Understanding the AppDynamics error detection life cycle can help you to troubleshoot missing errors and related error metrics. Use agent log files to search for evidence of each phase of error detection and reporting. Errors and exceptions are counted as Application Diagnostic Data metrics (ADDs) internally and they count towards the limits that are in place for ADDs. The phases in the error detection process are depicted in the following image: Phase 1- Error Detections Phase 2 - Error Registration Phase 3 - Error Metrics Registration Phase 4 - Error Metrics Reporting Each step must complete successfully before the next step begins. When you are troubleshooting, verify that each step happened successfully. Determining If Error Detection and Reporting Are Successful Use the Agent Log Files Generate and retrieve the agent logs using Debug level. You may need both the agent logs and the REST logs. Be sure to request the logs with logger level=debug. When you search, it is good to search all files at the same time for the best results. Registration of the higher-level objects such as BTs, backends, and errors will appear in the agent.log files. Metric registration and metric uploads will appear in the REST log. Phase 1 - Error Detection AppDynamics reports errors that occur during the execution of business transactions and error messages outside the context of a business transaction that are logged by the application server (called application server exceptions) Typical examples of errors are: Logged Exceptions or Messages Errors based on HTTP Return Codes Errors based on Redirect Pages   Conditions that result in error detection: Unhandled Exceptions An unhandled exception that occurs during the execution of a business transaction is reported as an error. Unhandled exceptions that occur during an exit call, for example, calls to databases, web services, or message queue servers. HTTP error codes See the Supported Environments and Versions for your app agent to determine if the loggers you use are recognized by default by AppDynamics. Phase 2 - Confirm Error Registration When an error is detected, the agent sends a registration request to the controller and the controller assigns an ID to the error. The ID is used for communication between the agent and controller to identify exactly which objects are being reported and stored in the controller database. To confirm error registration, get the agent log files and look for the registration request and response entries and find the ID assigned to the error. 1. Find error registration request log entries. Use search string = "Sending ADDs to register" Search the log files using the search string Sending ADDs to register. You should see entries similar to this one for the initial error registration request: [AD Thread Pool-Global0] 07 Aug 2014 16:44:07,255 INFO ErrorProcessor - Sending ADDs to register [ApplicationDiagnosticData{key='org.apache.tomcat.dbcp.dbcp.SQLNestedException:java.sql.SQLException:', name=SQLNestedException : SQLException, diagnosticType=ERROR, configEntities=null, summary='org.apache.tomcat.dbcp.dbcp.SQLNestedException caused by java.sql.SQLException'}] The string between the brackets [] identifies the type of object (Application Diagnostic Data or ADD), and contains values for key, name, diagnosticType, and summary. In this case, the diagnosticType=ERROR, and the key=org.apache.tomcat.dbcp.dbcp.SQLNestedException:java.sql.SQLException. Using that specific error key as a search string, you can find the log entry that contains the ID assigned to this error. In this case the ID is 53 as shown in this log example. [AD Thread Pool-Global0] 07 Aug 2014 16:44:07,340  INFO ErrorProcessor - Error Objects registered with controller :{org.apache.tomcat.dbcp.dbcp.SQLNestedException:java.sql.SQLException:=53} 2. Find error registration response log entries Use search string = "Error Objects registered with controller" Alternately, you can use the more general search string "Error Objects registered with controller" to find all the error objects that were registered during the duration of your logging session. The search results below show multiple errors and their IDs highlighted with a red outline. If you do not see successful registration request and response, look in the Controller Logs for exceptions related to error registration. Phase 3 - Confirm Metric Registration After successful error registration, the agent keeps an error map in memory with the ID, type, and name for each error. The next time the error is detected, the agent is ready to report metrics. A similar process of request and response between the agent and controller enables the controller to assign each metric an ID. The errors per minute metrics are reported at the following levels of aggregation: Application level - Total counts for the error across the entire application. The log looks similar to:  BTM|Application Diagnostic Data|Error:61|Errors per Minute Tier level - Total counts for the error across the tier. The log looks similar to: BTM|Application Summary|Component:1|Errors per Minute BTM|Application Summary|Component:2|Exit Call:HTTP|To:3|Errors per Minute Business transaction level - Totals by BT and exit calls:  BTM|BTs|BT:101|Component:1|Errors per Minute BTM|BTs|BT:102|Component:1|Exit Call:JDBC|To:{[UNRESOLVED][4]}|Errors per Minute Note: In the logs, Component=tier. Find error registration request log entries. Use search string = Application Diagnostic Data Using this search string should give search results similar to the following: Line 1366: <metric time-rollup-type="AVERAGE" name="BTM|Application Diagnostic Data|Error:61|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> Line 1392: <metric time-rollup-type="AVERAGE" name="BTM|Application Diagnostic Data|Error:71|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> Line 1402: <metric time-rollup-type="AVERAGE" name="BTM|Application Diagnostic Data|Error:53|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> Line 1403: <metric time-rollup-type="AVERAGE" name="BTM|Application Diagnostic Data|Error:70|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> ... Line 1561: <metric id="2450" name="BTM|Application Diagnostic Data|Error:61|Errors per Minute"/> Line 1575: <metric id="2539" name="BTM|Application Diagnostic Data|Error:71|Errors per Minute"/> Line 1515: <metric id="2463" name="BTM|Application Diagnostic Data|Error:53|Errors per Minute"/> Line 1568: <metric id="2546" name="BTM|Application Diagnostic Data|Error:70|Errors per Minute"/> Use search string = Errors per Minute Results similar to the following will show metrics for the varying levels of rollup indicated in BOLD: Line 1394: <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:131|Component:7|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> Line 1395: <metric time-rollup-type="AVERAGE" name="BTM|Application Summary|Component:7|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> Line 1402: <metric time-rollup-type="AVERAGE" name="BTM|Application Diagnostic Data|Error:53|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> Line 1403: <metric time-rollup-type="AVERAGE" name="BTM|Application Diagnostic Data|Error:70|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> Line 1442: <metric time-rollup-type="AVERAGE" name="BTM|Application Diagnostic Data|Error:69|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> Line 1445: <metric time-rollup-type="AVERAGE" name="BTM|Backends|Component:{[UNRESOLVED][13]}|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> Line 1460: <metric time-rollup-type="AVERAGE" name="BTM|Application Diagnostic Data|Error:76|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> ... Line 1536: <metric id="2475" name="BTM|Backends|Component:{[UNRESOLVED][13]}|Errors per Minute"/> Line 1546: <metric id="2342" name="BTM|BTs|BT:131|Component:7|Errors per Minute"/> Line 1491: <metric id="2512" name="BTM|Application Summary|Component:7|Errors per Minute"/> Line 1515: <metric id="2463" name="BTM|Application Diagnostic Data|Error:53|Errors per Minute"/> Line 1531: <metric id="2456" name="BTM|Application Diagnostic Data|Error:69|Errors per Minute"/> Line 1496: <metric id="2521" name="BTM|Application Diagnostic Data|Error:76|Errors per Minute"/> Line 1568: <metric id="2546" name="BTM|Application Diagnostic Data|Error:70|Errors per Minute"/> 2. Find error registration response log entries. In the above search results, you can see both the metric registration request and the response containing the metric ID for several errors at the application level. The entries beginning with "<metric id=" show the ID assigned to each separate error's metrics. You can see the metric ID assigned at line 1515 to Error:53 is 2463. If you do not see successful request and response for the metric, look in the Controller Logs for exceptions related to error metric registration. Phase 4 - Error Metrics Reporting After successful metric registration, metrics are reported to the controller every minute. Using the metric ID, you can search the REST log for the metric upload.  Find error metric upload log entries: Use the metric ID that you found when you verified metric registration. Look for log entries showing error metric reporting uploads. This example shows search results from using the metric ID value of "2463" to search the REST logs: Line 1515: <metric id="2463" name="BTM|Application Diagnostic Data|Error:53|Errors per Minute"/> Line 1686: <metric id='2463', value[sum=1, count=1, min=1, max=1, current=1]> Line 2511: <metric id='2463', value[sum=0, count=1, min=0, max=0, current=0]> Line 3207: <metric id='2463', value[sum=0, count=1, min=0, max=0, current=0]> Line 3919: <metric id='2463', value[sum=0, count=1, min=0, max=0, current=0]> Line 4578: <metric id='2463', value[sum=0, count=1, min=0, max=0, current=0]> If you do not see successful error metric data uploads, look in the Controller Logs for exceptions related to metric data upload. Notes on Error Registration Counts and ADD limits Application Diagnostic Data metrics include errors, exceptions, and async threads (and a few other things, such as snapshots). Error/Thread ADD registration limits counts behave as follows: If an Error/Thread ADD is deleted or excluded the relevant ADD registration limits count are decremented. If an Error/Thread ADD is unexcluded the relevant ADD registration limits count are incremented. Changed the exclude/unexclude operation so that you can not exclude an already excluded ADD or unexclude an already unexcluded ADD. This keeps ADD registration limit counts in a good state even if the exclude/unexclude APIs are used incorrectly. Note that limiting ERROR ADD registration prevents STACK_TRACE ADD registration as well. When a tier is deleted, the Error/Thread ADD registration limits counts are decremented accordingly.
Understanding the AppDynamics asynchronous (async) thread detection life cycle can help you to troubleshoot missing threads and related async metrics. Use agent log files to search for evidence of ea... See more...
Understanding the AppDynamics asynchronous (async) thread detection life cycle can help you to troubleshoot missing threads and related async metrics. Use agent log files to search for evidence of each phase of thread detection and reporting. The phases in the thread detection process are described here: AppDynamics Auto Discovery Life Cycle. Each step must complete successfully before the next step begins. When you are troubleshooting, verify that each step happened successfully. Determining Successful Thread Detection and Reporting Async Thread Detection Confirm Thread Registration Confirm Metric Registration Thread Metrics Reporting Determining Successful Thread Detection and Reporting  Generate and retrieve the agent logs using Debug level. For details on how to do this, see Enable Debug Level Logging. You may need both the agent logs and the REST logs. If so, see Request Agent Log Files. Be sure to request the logs with logger level=debug. When you search, it is efficient to search all files at the same time for the best results. Registration of the higher-level objects such as BTs, backends, and errors appear in the agent.log files. Metric registration and metric uploads appear in the REST log. 1. Async Thread Detection For details on how AppDynamics handles multi-threaded transactions, see: Trace Multithreaded Transactions for Java Monitor Async Backends for .NET You can verify instrumentation in the agent and BCT log. For example, this shows instrumentation of the run() and init() methods for the CartAction$1 thread. Agent Log showing instrumentation agent.2013_08_01__20_05_55.0.log:[AD Thread Pool-Global1] 01 Aug 2013 20:12:31,083 INFO AClassTransformation - Re-transforming class [com.appdynamicspilot.action.CartAction$1] because of newly added rule [Runtime [interface=SM{ex_type=EQUALS, ex_pattern='java.lang.Runnable', type=EQUALS, pattern='java.lang.Runnable'}], Method [ method=SM{ex_type=EQUALS, ex_pattern='run', type=EQUALS, pattern='run'}] ] agent.2013_08_01__20_05_55.0.log:[AD Thread Pool-Global1] 01 Aug 2013 20:12:31,443 INFO AClassTransformation - Re-transforming class [com.appdynamicspilot.action.CartAction$1] because of newly added rule [Runtime [interface=SM{ex_type=EQUALS, ex_pattern='java.lang.Runnable', type=EQUALS, pattern='java.lang.Runnable'}], Method [ method=SM{ex_type=EQUALS, ex_pattern='<init>', type=EQUALS, pattern='<init>'}] ] BCT Log showing interceptor Matching method <init> ((Lcom/appdynamicspilot/action/CartAction;ILcom/appdynamicspilot/jms/MessageProducer;Lcom/appdynamicspilot/service/CartService;)V) Applying method interceptor async.handoffAsyncHandOffIdentificationTracker at com/appdynamicspilot/action/CartAction$1.<init> ((Lcom/appdynamicspilot/action/CartAction;ILcom/appdynamicspilot/jms/MessageProducer;Lcom/appdynamicspilot/service/CartService;)V) id:95 Matching method run (()V) Applying method interceptor async.handoffAsyncHandOffExecutionTracker at com/appdynamicspilot/action/CartAction$1.run (()V) id:96 2. Confirm Thread Registration When a thread is detected, the agent sends a registration request to the controller and the controller assigns an ID to the thread. The ID is used for communication between the agent and controller to identify exactly which objects are being reported and stored in the controller database. You can find the thread registration log entries in the agent.log files. To confirm registration, look for the registration request and response entries and find the ID assigned to the thread. To find thread registration request log entries, search the log files using the search string, "INFO AsyncCorrelationADDRepository - Sending ADDs to register". You should see entries similar to this: Thread Registration Request [AD Thread Pool-Global0] 03 Aug 2013 00:17:17,075  INFO AsyncCorrelationADDRepository - Sending ADDs to register [ApplicationDiagnosticData{key='CartAction$1', name=CartAction$1, diagnosticType=THREAD_TASK, configEntities=null, summary='CartAction$1'}] The string between the brackets [] identifies the type of object (Application Diagnostic Data or ADD), and contains values for key, name, diagnosticType, and summary. In this case, the diagnosticType=THREAD_TASK, and the key=CartAction$1. Using "CartAction$1" as the search string, you can find the log entry that contains the ID assigned to this error. In this case the ID is 75 as shown in here. Thread Registration Response [AD Thread Pool-Global0] 03 Aug 2013 00:17:07,054  INFO AsyncCorrelationADDRepository - Registered ADDs [{CartAction$1=75}] Find thread registration response log entries Use search string = INFO AsyncCorrelationADDRepository - Registered ADDs Use the more general search string "INFO AsyncCorrelationADDRepository - Registered ADDs" to find all the thread objects that were registered during the duration of your logging session.  If you do not see successful registration request and response, look in the Controller Logs for exceptions related to error registration. 3. Confirm Metric Registration After successful registration, the agent keeps an map in memory with the ID, type, and name for each thread. The next time the thread is detected, the agent is ready to report metrics. A similar process of request and response between the agent and controller enables the controller to assign each metric an ID. Find thread metric registration request log entries Use search string = Th:<thread-ID> Use the thread number identified in the previous step to find the metric registration requests and responses. Using the search string and substituting 75 for the thread ID (found in phase 2), gives search results similar to the following: <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:132|Component:7|Th:75|Average Response Time (ms)"hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="INDIVIDUAL"/> <metric time-rollup-type="SUM" name="BTM|BTs|BT:132|Component:7|Th:75|Number of Very Slow Calls" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:132|Component:7|Th:75|Exit Call:JMS|To:{[UNRESOLVED][13]}|Average Response Time (ms)" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="INDIVIDUAL"/> <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:132|Component:7|Th:75|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|Application Summary|Component:7|Th:75|Calls per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|Application Summary|Component:7|Th:75|Exit Call:JMS|To:{[UNRESOLVED][13]}|Calls per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="SUM" name="BTM|BTs|BT:132|Component:7|Th:75|Number of Slow Calls" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|Application Summary|Component:7|Th:75|Exit Call:JMS|To:{[UNRESOLVED][13]}|Average Response Time (ms)" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="INDIVIDUAL"/> <metric time-rollup-type="SUM" name="BTM|BTs|BT:132|Component:7|Th:75|Stall Count" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:132|Component:7|Th:75|Calls per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|Application Summary|Component:7|Th:75|Exit Call:JMS|To:{[UNRESOLVED][13]}|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:132|Component:7|Th:75|Exit Call:JMS|To:{[UNRESOLVED][13]}|Calls per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:132|Component:7|Th:75|Exit Call:JMS|To:{[UNRESOLVED][13]}|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:132|Component:7|Th:75|Normal Average Response Time (ms)" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="INDIVIDUAL"/> Find thread registration response log entries In the above search results, you can see both the metric registration request and the response. The entries beginning with "<metric id=" show the ID assigned to each separate error's metrics similar to this: Line 1475: <metric id="2445" name="BTM|BTs|BT:132|Component:7|Th:75|Normal Average Response Time (ms)"/> Line 1488: <metric id="2522" name="BTM|BTs|BT:132|Component:7|Th:75|Stall Count"/> Line 1494: <metric id="2502" name="BTM|BTs|BT:132|Component:7|Th:75|Errors per Minute"/> Line 1495: <metric id="2416" name="BTM|BTs|BT:132|Component:7|Th:75|Calls per Minute"/> Line 1497: <metric id="2552" name="BTM|Application Summary|Component:7|Th:75|Calls per Minute"/> Line 1522: <metric id="2529" name="BTM|BTs|BT:132|Component:7|Th:75|Exit Call:JMS|To:{[UNRESOLVED][13]}|Calls per Minute"/> Line 1537: <metric id="2496" name="BTM|Application Summary|Component:7|Th:75|Exit Call:JMS|To:{[UNRESOLVED][13]}|Calls per Minute"/> ... You can see the metric ID assigned to each metric for thread ID=75. Note the BT (business transaction) ID=132.  If you do not see successful request and response for the metric, look in the Controller Logs for exceptions related to error metric registration. 4. Thread Metrics Reporting After successful metric registration, metrics are reported to the controller every minute. Using the metric ID, you can search the REST log for the metric upload.  Find thread metric upload log entries: Use the metric ID that you found at prior step when you verified metric registration. Look for log entries showing thread metric reporting uploads. This example shows some of the search results matching the metric ID values in the previous step: <metric id='2445', value[sum=0, count=0, min=0, max=0, current=0]> <metric id='2522', value[sum=0, count=1, min=0, max=0, current=0]> <metric id='2502', value[sum=909, count=1, min=909, max=909, current=909]> If you do not see successful metric data uploads, look in the Controller Logs for exceptions related to metric data upload. Sample Logs A zip file containing the log files used for this topic is attached.
What are the steps to enabling DEBUG mode for logging? Updated on 7/26/18 Most AppDynamics app agents use INFO level logging by default. Sometimes, it is useful to collect logs with DEBUG level ... See more...
What are the steps to enabling DEBUG mode for logging? Updated on 7/26/18 Most AppDynamics app agents use INFO level logging by default. Sometimes, it is useful to collect logs with DEBUG level logging—for example: when confirming life cycle phases to debug configuration issues, or when AppDynamics Support requests DEBUG logs.  Once DEBUG mode is enabled, apply load on the application and collect the debug logs.  Table of Contents How do I enable DEBUG  level logging?  Related Resources  How do I enable DEBUG  level logging? On the Agents tab of the Nodes dashboard, scroll to the Agent Operations panel. Click Request Agent Log Files.  By default the agent zips all logs in the agent log directory into one zip file. For the Java Agent or PHP Agent, you can select one of the other options: Output from a specific logger at a set level, for a fixed duration: use com.singularity and select Logger Level=Debug, Specify a duration, such as 15 minutes.  (Optional) Enter a name for your request. The name you provide identifies your request in the name of the generated zip file. For example, the name CurrentTime produces a zip file named similar to CurrentTime_Node_8000_1390503141046.zip. After the logging session completes and the log file has been written to disk, click Request Agent Log Files and the logged request and response output will appear in the Controller UI. You must Request the log files before you can download them. Related Resources How do I enable EUM Client debug logging dynamically in the Controller? How do I enable debug logging for an on-premise EUM Server or EUM Processor? How do I enable EUM debug logging for the .NET app agent? How do I enable debug logging for Platform Admin tool?
ISSUE: Controller IP changed but the .NET Agent doesn't pick up the DNS Change The agents are configured to use a hostname currently to communicate with the Controller. The Controller's IP address... See more...
ISSUE: Controller IP changed but the .NET Agent doesn't pick up the DNS Change The agents are configured to use a hostname currently to communicate with the Controller. The Controller's IP address and DNS entry are being changed, for example in the case that you are migrating your Controller to a new subnet.   Background By default, the .NET agent leverages the .NET Framework mechanism to do DNS resolution and connection handling. The issue here is that the DNS entry is cached for the period of time set in the TTL for the DNS entry.  As a result, even after a DNS update, there is a chance for failed communication until the TTL expires and the DNS entry gets updated. This is not specific to the .NET Agent but .NET in general. Because the .NET Agent high availability configuration enables the agent to detect this situation and force a flush of the of the DNS cache, you can take advantage of HA configuration to address this issue. The agent doesn't enable high availability by default, therefore this requires a configuration change and a restart of the AppDynamics.Agent.Coordinator service for the change to take effect. Then, after the next AppPool recycle the agent will pick up the new DNS information correctly. Because we flush the DNS cache every 5 minutes, the max delay for the change to be visible is 5 minutes.  Resolution To enable the correct failover for the agent in the config.xml, set the high_availability attribute to true in the config.xml file similar to the following: <?xml version="1.0" encoding="utf-8"?> <appdynamics-agent xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <controller host="your_host" port="8080" ssl="false" high_availability="true"> <application name="your_app" /> </controller> <machine-agent /> <app-agents> <IIS> <automatic /> <applications /> </IIS> </app-agents> </appdynamics-agent>
Updated 6/15/18 If one or more of your expected business transactions (BTs) are not appearing on the Business Transaction list in your Controller UI, complete the following troubleshooting steps t... See more...
Updated 6/15/18 If one or more of your expected business transactions (BTs) are not appearing on the Business Transaction list in your Controller UI, complete the following troubleshooting steps to determine the root cause. Do you see any business transactions? Has the BT limit been reached? Do you need to revise your BT detection configuration? Are the application agents reporting? 1. Do you see any business transactions? Check the Business Transaction list in your Controller UI, which will display only business transactions that have performance data in the selected time range by default. If you are missing data, you may need to modify your filters in case some BTs are hidden by the filter view options: View Business Transactions. Your application may also be making calls that are not hitting the default supported interceptors. Alternatively, it could be making calls directly to EJB Entry Points or Spring Bean Entry Points, for which we disable auto discovery by default. 2. Has the BT Limit Been Reached? AppDynamics limits the number of BTs to 50 transactions per agent and 200 transactions per business application. If traffic exceeds the 50 BTs per agent limit, any additionally detected BTs will be captured and aggregated into a virtual business transaction called "All Other Traffic." There is one "All Other Traffic" BT for each tier on which the limit is reached. Check if the expected BTs have been categorized as “All Other Traffic” by opening the dashboard for the “All Other Traffic - <tier name>” BT and clicking the View Traffic Details link. From here, you can see a list of entry points that were hit by incoming requests after the registration limit was maxed out. If needed, you can move transactions from the All Other Traffic bucket. 3. Do you need to revise your BT detection configuration? An important part of implementing AppDynamics is customizing your BT detection configuration so you can capture the most relevant transactions for your business. In addition to using automatic transaction discovery rules, you can use custom match rules  for more granular control. If customizations aren’t configured properly, you may not be capturing all of your expected BTs. For instructions on customizing detection rules, review the resources below: Transaction Detection Rules .NET Business Transaction Detection Java Business Transaction Detection Custom Exclude Rule Examples 4. Are the application agents reporting?   If your agents aren’t reporting business transaction data, there may be a connection issue between your agents and the Controller. Check your agent-to-Controller connections to see if this is the root cause for missing data: Verify agent-controller connection. For additional troubleshooting tips, visit: App Agent Not Reporting.
This flowchart describes the steps to confirm BT instrumentation when you are using a web framework. For a complete description of the BT Life Cycle, see: Business Transaction Discovery Life Cycle. ... See more...
This flowchart describes the steps to confirm BT instrumentation when you are using a web framework. For a complete description of the BT Life Cycle, see: Business Transaction Discovery Life Cycle. Supported Framework? First check our lists of supported environments to see if your framework is one that is discovered automatically by your agent: App Server Agents Supported Environments 1. Review the BCT Log Retrieve the Bytecode Transformer Log and look for entry point interceptors starting with entry.<your_framework> - for example, "entry.servlet" or "entry.struts". See Agent Log Files for details on requesting the agent logs. 2. Look for Entry Point Exceptions If you find evidence in the BCT log that the entry point was instrumented, look in the agent logs for entry point exceptions. See Request Agent Log Files for details on requesting agent logs.  If you find exceptions for your entry point, contact AppDynamics support for additional assistance.  If you do not see exceptions, review this article Business Transaction Discovery Life Cycle and use the steps at Phase 2 to verify registration with the controller. 3. Run Find Entry Points Use the procedure here Find Entry Points 4 Configure POJO or POCO Custom Match Rule For details see POJO Entry Points or POCO Entry Points. 5. Contact AppDynamics Support http://www.appdynamics.com/support/ 6. Verifying BT Registration If your entry point was instrumented and there were no related entry point exceptions, you can move to phase 2 - BT registration: Business Transaction Discovery Life Cycle.
To troubleshoot business transaction discovery it is helpful to have an understanding of the AppDynamics entity life cycle. A business transaction is a discrete piece of business application function... See more...
To troubleshoot business transaction discovery it is helpful to have an understanding of the AppDynamics entity life cycle. A business transaction is a discrete piece of business application functionality that is tracked from beginning to end, potentially across multiple tiers of processing, to measure the time it takes to complete the functionality. Phase 1: Business Transaction Discovery The discovery of a business transaction (BT) is accomplished by detecting where the business transaction begins and instrumenting its entry point. An entry point is the specific class and method (Java and .NET), operation or URI that kicks off the discrete bit of functionality that we want to measure. The exact entry points that are instrumented are agent dependent.  For details on dynamic language agents, see the supported environments pages for each agent in the product documentation: Supported Environments and Versions. Where does a BT begin? Where the core business logic is processed and executed. It might not necessarily be the first application component in the path of the request. A Java Servlet that handles a complex HTTP request is a good example of an entry point. In some cases the Servlet might merely forward the request to another component such as an EJB Bean that has the core business logic. In this case the EJB Bean is the right place to define the transaction as it is easier to differentiate one transaction from another based on the EJB Bean executing it. For PHP the entry point is typically a URI or a page callback name, MVC controller action, page template name, or page template. For Node.js the entry point is typically a method, operation or a URI.  Was BT Discovery Successful? Use the following steps: If you see BTs, but some that you expect to see are missing, first check that the BT limit was not exceeded. Was the BT limit reached? See All Other Traffic Business Transaction. If the BT limit was not reached or you do not find evidence of the business transaction in the traffic details of the overflow BT, go to the next step. Are you using a web framework? Yes, go to Confirming BT Instrumentation for Web Frameworks Are you running a background task? Yes, go to Confirming BT Instrumentation for Batch Frameworks If your entry point was not instrumented (see steps 3 and 4) and find entry points did not reveal a useful entry point, consider contacting AppDynamids Support at: http://www.appdynamics.com/support/. If you find evidence in the BCT log that the entry point was instrumented, check for entry point exceptions.        a. Look in the agent logs for entry point exceptions. For details on retrieving the logs, see Request Agent Log Files.        b. If you find exceptions for your entry point, contact support for additional assistance.        c. If you do not find any exceptions, move on to phase 2, registration. Phase 2: BT Registration When an entry point is successfully instrumented, it is assigned a name based on the rules for the entry point type or the custom naming rule (if applicable). Every BT is stored in the BT table in the controller database. The agent detects when the entry point is invoked and sends a BT registration request to the controller. Registration is when each BT is assigned an ID by the controller. After the ID is assigned, communication between the agent and controller uses the ID to identify exactly which BTs are being reported and stored in the controller database. You can review the BT registration log entries in the REST log. Use the following steps to confirm BT registration: 1. Generate and retrieve the REST log. For details on how to do this, see Request Agent Log Files. Look for registration and response details. In the following ViewCart BT request, you can see the entry point type is STRUTS_ACTION, the internal name is ViewCart.sendItems. At this point the id=0 and name=null. [AD Thread Pool-Global1|AD Thread Pool-Global1] 20 Nov 2012 09:49:44,491 DEBUG BusinessTransactionRegistry - REST - Sending Binary request - URL \[ /controller/instance/2/btregistration\] Payload \-****\**\**BT Registration Data*\******\* ... <AConfigObject ( AConfigObject ( com.singularity.ee.controller.api.dto.transactionmonitor.transactiondefinition.BusinessTransaction@8f000904 id = 0 name = null ) applicationId = 0 ) [ id=0, entryPointType=STRUTS_ACTION, internalName=ViewCart.sendItems, componentId=2, applicationComponentName=null, matchCriteria=com.singularity.ee.controller.api.dto.transactionmonitor.transactiondefinition.MatchCriteria@74cb7e2c, createdOn=null, background=false, configuration=null, createdNodeId=0, enabledForEUM=false, eumAutoEnablePossible=null]> In the following response you can see that the Controller assigned an id=80 and the name=ViewCart.sendItems. [AD Thread Pool-Global1] 20 Nov 2012 09:49:44,723 DEBUG BusinessTransactionRegistry - REST - Reading Binary payload - URL [ null] Size : [5376] Payload -********BT Registration Data******** ... <AConfigObject ( AConfigObject ( com.singularity.ee.controller.api.dto.transactionmonitor.transactiondefinition.BusinessTransaction@8f000904 id = 80 name = ViewCart.sendItems ) applicationId = 4 ) [ id=80, entryPointType=STRUTS_ACTION, internalName=ViewCart.sendItems, componentId=2, applicationComponentName=null, matchCriteria=com.singularity.ee.controller.api.dto.transactionmonitor.transactiondefinition.MatchCriteria@2ea28708, createdOn=null, background=false, configuration=null, createdNodeId=3, enabledForEUM=false, eumAutoEnablePossible=null]> 2. If you do not see a successful registration request and response, look in the Controller Logs for related exceptions. Phase 3 - BT metrics registration After successful BT registration, the agent keeps a BT map in memory with the ID, type, and name for each BT. The next time the entry point is hit, the agent is ready to report metrics. A similar process of request and response between the agent and controller occurs and the controller assigns each metric an ID. Use the following steps to confirm BT metric registration: Generate and retrieve the REST log. For details on how to do this, see Request Agent Log Files. Look for log entries showing metric registration. In the following request, you can see "BT:80", which we know from our review at phase 2 identifies the ViewCart.sendItems business transaction. The two metrics shown below represent Calls per Minute and Average Response Time for the ViewCart.sendItems BT. [AD Thread-Metric Reporter0] 20 Nov 2012 09:50:44,079 DEBUG MetricSender - Sending REST request - URL [ /controller/instance/2/metricregistration] XML payload -<request> ... <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:80|Component:2|Calls per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:80|Component:2|Average Response Time (ms)" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="INDIVIDUAL"/> In the following response you can see BT:80, identifying the ViewCart.sendItems BT. You can see that metric id=1815 was assigned to the Calls per Minute metric. A metric id=1813 was assigned to the Average Response Time metric. [AD Thread-Metric Reporter0] 20 Nov 2012 09:50:44,346 DEBUG MetricSender - REST Response Received -> <request> ... <metric id="1815" name="BTM|BTs|BT:80|Component:2|Calls per Minute"/> <metric id="1813" name="BTM|BTs|BT:80|Component:2|Average Response Time (ms)"/> 3.  If you do not see successful request and response for the metric, look in the Controller Logs for exceptions related to backend metric registration. Phase 4 - BT metrics reporting After successful, metric registration, BT metrics are reported to the controller every minute. Using the metric id, you can search the REST log for the metric upload. Use the following steps to confirm BT metric upload: 1. Generate and retrieve the REST log. For details on how to do this, see Request Agent Log Files. 2. Look for log entries showing metric reporting uploads similar to the following: [AD Thread-Metric Reporter0] 20 Nov 2012 09:50:44,361 DEBUG MetricSender - REST - Sending Binary request - URL [ /controller/instance/2/metrics] Payload -********Metric Data******** ... <metric id='1815', value[sum=32, count=1, min=32, max=32, current=32]> <metric id='1813', value[sum=8693, count=32, min=4, max=1614, current=53]> 3. If you do not see successful metric data uploads, look in the Controller Logs for related  exceptions. Phase 5 - BT metrics roll up App agents do not communicate with each other, they report metrics to the controller database. The controller aggregates the metrics from all nodes in each tier to present a unified overall picture in the Controller console. If you have verified that phases 1-4 of the BT discovery life cycle have been successful and you are not seeing the expected BT metrics, contact AppDynamics Support for assistance: http://www.appdynamics.com/support/.
Default JMX Metrics Example  Some MBeans exposed by various supported application servers are preconfigued as JMX Metrics that can be viewed in the JMX Metrics Browser. In this screen capture of... See more...
Default JMX Metrics Example  Some MBeans exposed by various supported application servers are preconfigued as JMX Metrics that can be viewed in the JMX Metrics Browser. In this screen capture of the JMX Metrics Browser, under Web Container Runtime, you can see the two metrics "Error Count" and "Request Count". Where did these come from? These metrics were configured from MBeans exposed by the Tomcat web container.  To see the configuration for these metrics, do the following: 1. Select Configure -> Instrumentation. 2. Select the application. 3. Select the JMX tab. In this "metric rule" named Tomcat_GlobalRequestProcessor", you can see where the Metric Path and the MBean matching criteria in the domain "Catalina" have been specified. This path in the metric browser was seen in the first screen capture. The next screen capture shows the two attributes of this MBean.