All Topics

Learn Splunk

All Topics

This topic describes the tier and node life cycle within AppDynamics. Phase 1 - Agent Startup Phase 2 - Registration Phase 3 - Metric Registration Phase 4 - Metric Reporting Phase 5 - Metri... See more...
This topic describes the tier and node life cycle within AppDynamics. Phase 1 - Agent Startup Phase 2 - Registration Phase 3 - Metric Registration Phase 4 - Metric Reporting Phase 5 - Metric Rollup A combination of agent logs and REST logs reveal the life cycle details for the tier and node metrics. Phase 1 - Agent Startup The application name, tier name, and node name are passed to the agent as part of the start-up process. For Java, the following JVM arguments are used: -Dappdynamics.agent.applicationName -Dappdynamics.agent.tierName -Dappdynamics.agent.nodeName Example log snippet [Thread-0] 21 May 2014 21:15:16,041 INFO AgentKernel - JVM Args : ... | -javaagent:/apps/appdynamics/current/javaagent.jar | -Dappdynamics.controller.hostName=appdynamics.<domain>.net | -Dappdynamics.controller.port=8090 | -Dappdynamics.agent.applicationName=CL-Tax-WL-prod | -Dappdynamics.agent.tierName=GoWeb2-prod | -Dappdynamics.agent.nodeName=GoWeb05-02-prod | -Dappdynamics.agent.runtime.dir=/apps/Java/GoWeb/appdynamics ... From these JVM arguments, you can see the application name is CL-Tax-WL-prod, tier name is GoWeb2-prod, and the node name is GoWeb05-02-prod. -Dappdynamics.agent.applicationName=CL-Tax-WL-prod | -Dappdynamics.agent.tierName=GoWeb2-prod | -Dappdynamics.agent.nodeName=GoWeb05-02-prod Phase 2 - Registration The agent sends the registration request to the controller to register the application, tier, and node. [Thread-0] 21 May 2014 21:15:17,220  INFO ConfigurationChannel - Sending Registration request with: Application Name [CL-Tax-WL-prod], Tier Name [GoWeb2-prod], Node Name [GoWeb05-02-prod], Host Name [fada1wap05] Node Unique Local ID [GoWeb05-02-prod], Version [Server Agent v3.6.1.0 GA #2012-12-11_14-59-42 r2346f2ff54e68a0be000f975f9d0ebc828f8b660 42] Controller Provides/Confirms the IDs The component ID is the internal designation for a tier.   [Thread-0] 21 May 2014 21:15:17,982 INFO ConfigurationChannel - Auto agent registration SUCCEEDED! [Thread-0] 21 May 2014 21:15:17,982 INFO ConfigurationChannel - Registration information received Node ID[153] Component ID[57] Application ID [16]   Phase 3 - Metric Registration The metrics are reported from each agent/node, and registered with the controller. Tier level metrics look like this in the logs: BTM|Application Summary|Component:10|Calls per Minute BTM|Application Summary|Component:10|Exit Call:HTTP|To:14|Calls per Minute BTM|Application Summary|Component:7|Exit Call:JDBC|To:{[UNRESOLVED][9]}|Calls per Minute Log entries from the agent logs for metric registration and reporting for these tier-level metrics: Calls per Minute Errors per Minute Stall Count Number of Slow Calls [AD Thread-Metric Reporter0] 21 Mar 2014 11:28:30,708 DEBUG MetricHandler - 0 BTM|Application Summary|Component:16|Calls per Minute 28 UNREGISTERED 612018265 [AD Thread-Metric Reporter0] 21 Mar 2014 11:28:30,707 DEBUG MetricHandler - 0 BTM|Application Summary|Component:16|Errors per Minute 0 UNREGISTERED 277551759 [AD Thread-Metric Reporter0] 21 Mar 2014 11:28:30,708 DEBUG MetricHandler - 0 BTM|Application Summary|Component:16|Stall Count 0 UNREGISTERED 2010399551 [AD Thread-Metric Reporter0] 21 Mar 2014 11:28:30,708 DEBUG MetricHandler - 0 BTM|Application Summary|Component:16|Number of Slow Calls 0 UNREGISTERED 1630292838 Metric IDs applied to the unregistered metrics: [AD Thread-Metric Reporter0] 21 Mar 2014 11:28:40,706 DEBUG MetricHandler - adding unregistered metric BTM|Application Summary|Component:16|Errors per Minute [AD Thread-Metric Reporter0] 21 Mar 2014 11:28:40,707 DEBUG MetricHandler - adding unregistered metric BTM|Application Summary|Component:16|Stall Count [AD Thread-Metric Reporter0] 21 Mar 2014 11:28:40,707 DEBUG MetricHandler - adding unregistered metric BTM|Application Summary|Component:16|Number of Slow Calls [AD Thread-Metric Reporter0] 21 Mar 2014 11:28:40,707 DEBUG MetricHandler - adding unregistered metric BTM|BTs|BT:241|Component:16|Errors per Minute The following logs show the assignment of the metric ID to specific metrics. These logs are found in the REST log files. <metric id="7527" name="BTM\|Application Summary\|Component:16\|Number of Slow Calls"/> <metric id="7499" name="BTM\|BTs\|BT:241\|Component:16\|Errors per Minute"/> <metric id="7526" name="BTM\|Application Summary\|Component:16\|Stall Count"/> <metric id="7530" name="BTM\|Application Summary\|Component:16\|Calls per Minute"/> Phase 4 - Metric Reporting Metrics reported using the metric IDs. These logs are from the REST log files. <metric id='7527', value\[sum=0, count=1, min=0, max=0, current=0\]> <metric id='7499', value\[sum=0, count=1, min=0, max=0, current=0\]> <metric id='7526', value\[sum=0, count=1, min=0, max=0, current=0\]> <metric id='7530', value\[sum=28, count=1, min=28, max=28, current=28\]> Phase 5 - Metric Rollup This stage happens at the controller. The metrics from all nodes are rolled up in the controller according to the values of time-rollup-type and cluster-rollup-type that was specified at the time of metric registration. The metric registration logs can be found in the REST log. They look similar to the following: <node-id>35</node-id> <agent-type>APP_AGENT</agent-type> <metric time-rollup-type="AVERAGE" name="BTM|Application Summary|Component:16|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|Application Summary|Component:16|Infrastructure Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="SUM" name="BTM|Application Summary|Number of Slow Calls" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="SUM" name="BTM|Application Summary|Stall Count" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="SUM" name="BTM|Application Summary|Component:16|Stall Count" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="SUM" name="BTM|Application Summary|Component:16|Number of Slow Calls" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="SUM" name="BTM|Application Summary|Number of Slow End to End Messages" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="COLLECTIVE"/>
Understanding the AppDynamics error detection life cycle can help you to troubleshoot missing errors and related error metrics. Use agent log files to search for evidence of each phase of error detec... See more...
Understanding the AppDynamics error detection life cycle can help you to troubleshoot missing errors and related error metrics. Use agent log files to search for evidence of each phase of error detection and reporting. Errors and exceptions are counted as Application Diagnostic Data metrics (ADDs) internally and they count towards the limits that are in place for ADDs. The phases in the error detection process are depicted in the following image: Phase 1- Error Detections Phase 2 - Error Registration Phase 3 - Error Metrics Registration Phase 4 - Error Metrics Reporting Each step must complete successfully before the next step begins. When you are troubleshooting, verify that each step happened successfully. Determining If Error Detection and Reporting Are Successful Use the Agent Log Files Generate and retrieve the agent logs using Debug level. You may need both the agent logs and the REST logs. Be sure to request the logs with logger level=debug. When you search, it is good to search all files at the same time for the best results. Registration of the higher-level objects such as BTs, backends, and errors will appear in the agent.log files. Metric registration and metric uploads will appear in the REST log. Phase 1 - Error Detection AppDynamics reports errors that occur during the execution of business transactions and error messages outside the context of a business transaction that are logged by the application server (called application server exceptions) Typical examples of errors are: Logged Exceptions or Messages Errors based on HTTP Return Codes Errors based on Redirect Pages   Conditions that result in error detection: Unhandled Exceptions An unhandled exception that occurs during the execution of a business transaction is reported as an error. Unhandled exceptions that occur during an exit call, for example, calls to databases, web services, or message queue servers. HTTP error codes See the Supported Environments and Versions for your app agent to determine if the loggers you use are recognized by default by AppDynamics. Phase 2 - Confirm Error Registration When an error is detected, the agent sends a registration request to the controller and the controller assigns an ID to the error. The ID is used for communication between the agent and controller to identify exactly which objects are being reported and stored in the controller database. To confirm error registration, get the agent log files and look for the registration request and response entries and find the ID assigned to the error. 1. Find error registration request log entries. Use search string = "Sending ADDs to register" Search the log files using the search string Sending ADDs to register. You should see entries similar to this one for the initial error registration request: [AD Thread Pool-Global0] 07 Aug 2014 16:44:07,255 INFO ErrorProcessor - Sending ADDs to register [ApplicationDiagnosticData{key='org.apache.tomcat.dbcp.dbcp.SQLNestedException:java.sql.SQLException:', name=SQLNestedException : SQLException, diagnosticType=ERROR, configEntities=null, summary='org.apache.tomcat.dbcp.dbcp.SQLNestedException caused by java.sql.SQLException'}] The string between the brackets [] identifies the type of object (Application Diagnostic Data or ADD), and contains values for key, name, diagnosticType, and summary. In this case, the diagnosticType=ERROR, and the key=org.apache.tomcat.dbcp.dbcp.SQLNestedException:java.sql.SQLException. Using that specific error key as a search string, you can find the log entry that contains the ID assigned to this error. In this case the ID is 53 as shown in this log example. [AD Thread Pool-Global0] 07 Aug 2014 16:44:07,340  INFO ErrorProcessor - Error Objects registered with controller :{org.apache.tomcat.dbcp.dbcp.SQLNestedException:java.sql.SQLException:=53} 2. Find error registration response log entries Use search string = "Error Objects registered with controller" Alternately, you can use the more general search string "Error Objects registered with controller" to find all the error objects that were registered during the duration of your logging session. The search results below show multiple errors and their IDs highlighted with a red outline. If you do not see successful registration request and response, look in the Controller Logs for exceptions related to error registration. Phase 3 - Confirm Metric Registration After successful error registration, the agent keeps an error map in memory with the ID, type, and name for each error. The next time the error is detected, the agent is ready to report metrics. A similar process of request and response between the agent and controller enables the controller to assign each metric an ID. The errors per minute metrics are reported at the following levels of aggregation: Application level - Total counts for the error across the entire application. The log looks similar to:  BTM|Application Diagnostic Data|Error:61|Errors per Minute Tier level - Total counts for the error across the tier. The log looks similar to: BTM|Application Summary|Component:1|Errors per Minute BTM|Application Summary|Component:2|Exit Call:HTTP|To:3|Errors per Minute Business transaction level - Totals by BT and exit calls:  BTM|BTs|BT:101|Component:1|Errors per Minute BTM|BTs|BT:102|Component:1|Exit Call:JDBC|To:{[UNRESOLVED][4]}|Errors per Minute Note: In the logs, Component=tier. Find error registration request log entries. Use search string = Application Diagnostic Data Using this search string should give search results similar to the following: Line 1366: <metric time-rollup-type="AVERAGE" name="BTM|Application Diagnostic Data|Error:61|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> Line 1392: <metric time-rollup-type="AVERAGE" name="BTM|Application Diagnostic Data|Error:71|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> Line 1402: <metric time-rollup-type="AVERAGE" name="BTM|Application Diagnostic Data|Error:53|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> Line 1403: <metric time-rollup-type="AVERAGE" name="BTM|Application Diagnostic Data|Error:70|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> ... Line 1561: <metric id="2450" name="BTM|Application Diagnostic Data|Error:61|Errors per Minute"/> Line 1575: <metric id="2539" name="BTM|Application Diagnostic Data|Error:71|Errors per Minute"/> Line 1515: <metric id="2463" name="BTM|Application Diagnostic Data|Error:53|Errors per Minute"/> Line 1568: <metric id="2546" name="BTM|Application Diagnostic Data|Error:70|Errors per Minute"/> Use search string = Errors per Minute Results similar to the following will show metrics for the varying levels of rollup indicated in BOLD: Line 1394: <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:131|Component:7|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> Line 1395: <metric time-rollup-type="AVERAGE" name="BTM|Application Summary|Component:7|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> Line 1402: <metric time-rollup-type="AVERAGE" name="BTM|Application Diagnostic Data|Error:53|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> Line 1403: <metric time-rollup-type="AVERAGE" name="BTM|Application Diagnostic Data|Error:70|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> Line 1442: <metric time-rollup-type="AVERAGE" name="BTM|Application Diagnostic Data|Error:69|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> Line 1445: <metric time-rollup-type="AVERAGE" name="BTM|Backends|Component:{[UNRESOLVED][13]}|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> Line 1460: <metric time-rollup-type="AVERAGE" name="BTM|Application Diagnostic Data|Error:76|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> ... Line 1536: <metric id="2475" name="BTM|Backends|Component:{[UNRESOLVED][13]}|Errors per Minute"/> Line 1546: <metric id="2342" name="BTM|BTs|BT:131|Component:7|Errors per Minute"/> Line 1491: <metric id="2512" name="BTM|Application Summary|Component:7|Errors per Minute"/> Line 1515: <metric id="2463" name="BTM|Application Diagnostic Data|Error:53|Errors per Minute"/> Line 1531: <metric id="2456" name="BTM|Application Diagnostic Data|Error:69|Errors per Minute"/> Line 1496: <metric id="2521" name="BTM|Application Diagnostic Data|Error:76|Errors per Minute"/> Line 1568: <metric id="2546" name="BTM|Application Diagnostic Data|Error:70|Errors per Minute"/> 2. Find error registration response log entries. In the above search results, you can see both the metric registration request and the response containing the metric ID for several errors at the application level. The entries beginning with "<metric id=" show the ID assigned to each separate error's metrics. You can see the metric ID assigned at line 1515 to Error:53 is 2463. If you do not see successful request and response for the metric, look in the Controller Logs for exceptions related to error metric registration. Phase 4 - Error Metrics Reporting After successful metric registration, metrics are reported to the controller every minute. Using the metric ID, you can search the REST log for the metric upload.  Find error metric upload log entries: Use the metric ID that you found when you verified metric registration. Look for log entries showing error metric reporting uploads. This example shows search results from using the metric ID value of "2463" to search the REST logs: Line 1515: <metric id="2463" name="BTM|Application Diagnostic Data|Error:53|Errors per Minute"/> Line 1686: <metric id='2463', value[sum=1, count=1, min=1, max=1, current=1]> Line 2511: <metric id='2463', value[sum=0, count=1, min=0, max=0, current=0]> Line 3207: <metric id='2463', value[sum=0, count=1, min=0, max=0, current=0]> Line 3919: <metric id='2463', value[sum=0, count=1, min=0, max=0, current=0]> Line 4578: <metric id='2463', value[sum=0, count=1, min=0, max=0, current=0]> If you do not see successful error metric data uploads, look in the Controller Logs for exceptions related to metric data upload. Notes on Error Registration Counts and ADD limits Application Diagnostic Data metrics include errors, exceptions, and async threads (and a few other things, such as snapshots). Error/Thread ADD registration limits counts behave as follows: If an Error/Thread ADD is deleted or excluded the relevant ADD registration limits count are decremented. If an Error/Thread ADD is unexcluded the relevant ADD registration limits count are incremented. Changed the exclude/unexclude operation so that you can not exclude an already excluded ADD or unexclude an already unexcluded ADD. This keeps ADD registration limit counts in a good state even if the exclude/unexclude APIs are used incorrectly. Note that limiting ERROR ADD registration prevents STACK_TRACE ADD registration as well. When a tier is deleted, the Error/Thread ADD registration limits counts are decremented accordingly.
Understanding the AppDynamics asynchronous (async) thread detection life cycle can help you to troubleshoot missing threads and related async metrics. Use agent log files to search for evidence of ea... See more...
Understanding the AppDynamics asynchronous (async) thread detection life cycle can help you to troubleshoot missing threads and related async metrics. Use agent log files to search for evidence of each phase of thread detection and reporting. The phases in the thread detection process are described here: AppDynamics Auto Discovery Life Cycle. Each step must complete successfully before the next step begins. When you are troubleshooting, verify that each step happened successfully. Determining Successful Thread Detection and Reporting Async Thread Detection Confirm Thread Registration Confirm Metric Registration Thread Metrics Reporting Determining Successful Thread Detection and Reporting  Generate and retrieve the agent logs using Debug level. For details on how to do this, see Enable Debug Level Logging. You may need both the agent logs and the REST logs. If so, see Request Agent Log Files. Be sure to request the logs with logger level=debug. When you search, it is efficient to search all files at the same time for the best results. Registration of the higher-level objects such as BTs, backends, and errors appear in the agent.log files. Metric registration and metric uploads appear in the REST log. 1. Async Thread Detection For details on how AppDynamics handles multi-threaded transactions, see: Trace Multithreaded Transactions for Java Monitor Async Backends for .NET You can verify instrumentation in the agent and BCT log. For example, this shows instrumentation of the run() and init() methods for the CartAction$1 thread. Agent Log showing instrumentation agent.2013_08_01__20_05_55.0.log:[AD Thread Pool-Global1] 01 Aug 2013 20:12:31,083 INFO AClassTransformation - Re-transforming class [com.appdynamicspilot.action.CartAction$1] because of newly added rule [Runtime [interface=SM{ex_type=EQUALS, ex_pattern='java.lang.Runnable', type=EQUALS, pattern='java.lang.Runnable'}], Method [ method=SM{ex_type=EQUALS, ex_pattern='run', type=EQUALS, pattern='run'}] ] agent.2013_08_01__20_05_55.0.log:[AD Thread Pool-Global1] 01 Aug 2013 20:12:31,443 INFO AClassTransformation - Re-transforming class [com.appdynamicspilot.action.CartAction$1] because of newly added rule [Runtime [interface=SM{ex_type=EQUALS, ex_pattern='java.lang.Runnable', type=EQUALS, pattern='java.lang.Runnable'}], Method [ method=SM{ex_type=EQUALS, ex_pattern='<init>', type=EQUALS, pattern='<init>'}] ] BCT Log showing interceptor Matching method <init> ((Lcom/appdynamicspilot/action/CartAction;ILcom/appdynamicspilot/jms/MessageProducer;Lcom/appdynamicspilot/service/CartService;)V) Applying method interceptor async.handoffAsyncHandOffIdentificationTracker at com/appdynamicspilot/action/CartAction$1.<init> ((Lcom/appdynamicspilot/action/CartAction;ILcom/appdynamicspilot/jms/MessageProducer;Lcom/appdynamicspilot/service/CartService;)V) id:95 Matching method run (()V) Applying method interceptor async.handoffAsyncHandOffExecutionTracker at com/appdynamicspilot/action/CartAction$1.run (()V) id:96 2. Confirm Thread Registration When a thread is detected, the agent sends a registration request to the controller and the controller assigns an ID to the thread. The ID is used for communication between the agent and controller to identify exactly which objects are being reported and stored in the controller database. You can find the thread registration log entries in the agent.log files. To confirm registration, look for the registration request and response entries and find the ID assigned to the thread. To find thread registration request log entries, search the log files using the search string, "INFO AsyncCorrelationADDRepository - Sending ADDs to register". You should see entries similar to this: Thread Registration Request [AD Thread Pool-Global0] 03 Aug 2013 00:17:17,075  INFO AsyncCorrelationADDRepository - Sending ADDs to register [ApplicationDiagnosticData{key='CartAction$1', name=CartAction$1, diagnosticType=THREAD_TASK, configEntities=null, summary='CartAction$1'}] The string between the brackets [] identifies the type of object (Application Diagnostic Data or ADD), and contains values for key, name, diagnosticType, and summary. In this case, the diagnosticType=THREAD_TASK, and the key=CartAction$1. Using "CartAction$1" as the search string, you can find the log entry that contains the ID assigned to this error. In this case the ID is 75 as shown in here. Thread Registration Response [AD Thread Pool-Global0] 03 Aug 2013 00:17:07,054  INFO AsyncCorrelationADDRepository - Registered ADDs [{CartAction$1=75}] Find thread registration response log entries Use search string = INFO AsyncCorrelationADDRepository - Registered ADDs Use the more general search string "INFO AsyncCorrelationADDRepository - Registered ADDs" to find all the thread objects that were registered during the duration of your logging session.  If you do not see successful registration request and response, look in the Controller Logs for exceptions related to error registration. 3. Confirm Metric Registration After successful registration, the agent keeps an map in memory with the ID, type, and name for each thread. The next time the thread is detected, the agent is ready to report metrics. A similar process of request and response between the agent and controller enables the controller to assign each metric an ID. Find thread metric registration request log entries Use search string = Th:<thread-ID> Use the thread number identified in the previous step to find the metric registration requests and responses. Using the search string and substituting 75 for the thread ID (found in phase 2), gives search results similar to the following: <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:132|Component:7|Th:75|Average Response Time (ms)"hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="INDIVIDUAL"/> <metric time-rollup-type="SUM" name="BTM|BTs|BT:132|Component:7|Th:75|Number of Very Slow Calls" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:132|Component:7|Th:75|Exit Call:JMS|To:{[UNRESOLVED][13]}|Average Response Time (ms)" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="INDIVIDUAL"/> <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:132|Component:7|Th:75|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|Application Summary|Component:7|Th:75|Calls per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|Application Summary|Component:7|Th:75|Exit Call:JMS|To:{[UNRESOLVED][13]}|Calls per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="SUM" name="BTM|BTs|BT:132|Component:7|Th:75|Number of Slow Calls" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|Application Summary|Component:7|Th:75|Exit Call:JMS|To:{[UNRESOLVED][13]}|Average Response Time (ms)" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="INDIVIDUAL"/> <metric time-rollup-type="SUM" name="BTM|BTs|BT:132|Component:7|Th:75|Stall Count" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:132|Component:7|Th:75|Calls per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|Application Summary|Component:7|Th:75|Exit Call:JMS|To:{[UNRESOLVED][13]}|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:132|Component:7|Th:75|Exit Call:JMS|To:{[UNRESOLVED][13]}|Calls per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:132|Component:7|Th:75|Exit Call:JMS|To:{[UNRESOLVED][13]}|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:132|Component:7|Th:75|Normal Average Response Time (ms)" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="INDIVIDUAL"/> Find thread registration response log entries In the above search results, you can see both the metric registration request and the response. The entries beginning with "<metric id=" show the ID assigned to each separate error's metrics similar to this: Line 1475: <metric id="2445" name="BTM|BTs|BT:132|Component:7|Th:75|Normal Average Response Time (ms)"/> Line 1488: <metric id="2522" name="BTM|BTs|BT:132|Component:7|Th:75|Stall Count"/> Line 1494: <metric id="2502" name="BTM|BTs|BT:132|Component:7|Th:75|Errors per Minute"/> Line 1495: <metric id="2416" name="BTM|BTs|BT:132|Component:7|Th:75|Calls per Minute"/> Line 1497: <metric id="2552" name="BTM|Application Summary|Component:7|Th:75|Calls per Minute"/> Line 1522: <metric id="2529" name="BTM|BTs|BT:132|Component:7|Th:75|Exit Call:JMS|To:{[UNRESOLVED][13]}|Calls per Minute"/> Line 1537: <metric id="2496" name="BTM|Application Summary|Component:7|Th:75|Exit Call:JMS|To:{[UNRESOLVED][13]}|Calls per Minute"/> ... You can see the metric ID assigned to each metric for thread ID=75. Note the BT (business transaction) ID=132.  If you do not see successful request and response for the metric, look in the Controller Logs for exceptions related to error metric registration. 4. Thread Metrics Reporting After successful metric registration, metrics are reported to the controller every minute. Using the metric ID, you can search the REST log for the metric upload.  Find thread metric upload log entries: Use the metric ID that you found at prior step when you verified metric registration. Look for log entries showing thread metric reporting uploads. This example shows some of the search results matching the metric ID values in the previous step: <metric id='2445', value[sum=0, count=0, min=0, max=0, current=0]> <metric id='2522', value[sum=0, count=1, min=0, max=0, current=0]> <metric id='2502', value[sum=909, count=1, min=909, max=909, current=909]> If you do not see successful metric data uploads, look in the Controller Logs for exceptions related to metric data upload. Sample Logs A zip file containing the log files used for this topic is attached.
What are the steps to enabling DEBUG mode for logging? Updated on 7/26/18 Most AppDynamics app agents use INFO level logging by default. Sometimes, it is useful to collect logs with DEBUG level ... See more...
What are the steps to enabling DEBUG mode for logging? Updated on 7/26/18 Most AppDynamics app agents use INFO level logging by default. Sometimes, it is useful to collect logs with DEBUG level logging—for example: when confirming life cycle phases to debug configuration issues, or when AppDynamics Support requests DEBUG logs.  Once DEBUG mode is enabled, apply load on the application and collect the debug logs.  Table of Contents How do I enable DEBUG  level logging?  Related Resources  How do I enable DEBUG  level logging? On the Agents tab of the Nodes dashboard, scroll to the Agent Operations panel. Click Request Agent Log Files.  By default the agent zips all logs in the agent log directory into one zip file. For the Java Agent or PHP Agent, you can select one of the other options: Output from a specific logger at a set level, for a fixed duration: use com.singularity and select Logger Level=Debug, Specify a duration, such as 15 minutes.  (Optional) Enter a name for your request. The name you provide identifies your request in the name of the generated zip file. For example, the name CurrentTime produces a zip file named similar to CurrentTime_Node_8000_1390503141046.zip. After the logging session completes and the log file has been written to disk, click Request Agent Log Files and the logged request and response output will appear in the Controller UI. You must Request the log files before you can download them. Related Resources How do I enable EUM Client debug logging dynamically in the Controller? How do I enable debug logging for an on-premise EUM Server or EUM Processor? How do I enable EUM debug logging for the .NET app agent? How do I enable debug logging for Platform Admin tool?
ISSUE: Controller IP changed but the .NET Agent doesn't pick up the DNS Change The agents are configured to use a hostname currently to communicate with the Controller. The Controller's IP address... See more...
ISSUE: Controller IP changed but the .NET Agent doesn't pick up the DNS Change The agents are configured to use a hostname currently to communicate with the Controller. The Controller's IP address and DNS entry are being changed, for example in the case that you are migrating your Controller to a new subnet.   Background By default, the .NET agent leverages the .NET Framework mechanism to do DNS resolution and connection handling. The issue here is that the DNS entry is cached for the period of time set in the TTL for the DNS entry.  As a result, even after a DNS update, there is a chance for failed communication until the TTL expires and the DNS entry gets updated. This is not specific to the .NET Agent but .NET in general. Because the .NET Agent high availability configuration enables the agent to detect this situation and force a flush of the of the DNS cache, you can take advantage of HA configuration to address this issue. The agent doesn't enable high availability by default, therefore this requires a configuration change and a restart of the AppDynamics.Agent.Coordinator service for the change to take effect. Then, after the next AppPool recycle the agent will pick up the new DNS information correctly. Because we flush the DNS cache every 5 minutes, the max delay for the change to be visible is 5 minutes.  Resolution To enable the correct failover for the agent in the config.xml, set the high_availability attribute to true in the config.xml file similar to the following: <?xml version="1.0" encoding="utf-8"?> <appdynamics-agent xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <controller host="your_host" port="8080" ssl="false" high_availability="true"> <application name="your_app" /> </controller> <machine-agent /> <app-agents> <IIS> <automatic /> <applications /> </IIS> </app-agents> </appdynamics-agent>
Updated 6/15/18 If one or more of your expected business transactions (BTs) are not appearing on the Business Transaction list in your Controller UI, complete the following troubleshooting steps t... See more...
Updated 6/15/18 If one or more of your expected business transactions (BTs) are not appearing on the Business Transaction list in your Controller UI, complete the following troubleshooting steps to determine the root cause. Do you see any business transactions? Has the BT limit been reached? Do you need to revise your BT detection configuration? Are the application agents reporting? 1. Do you see any business transactions? Check the Business Transaction list in your Controller UI, which will display only business transactions that have performance data in the selected time range by default. If you are missing data, you may need to modify your filters in case some BTs are hidden by the filter view options: View Business Transactions. Your application may also be making calls that are not hitting the default supported interceptors. Alternatively, it could be making calls directly to EJB Entry Points or Spring Bean Entry Points, for which we disable auto discovery by default. 2. Has the BT Limit Been Reached? AppDynamics limits the number of BTs to 50 transactions per agent and 200 transactions per business application. If traffic exceeds the 50 BTs per agent limit, any additionally detected BTs will be captured and aggregated into a virtual business transaction called "All Other Traffic." There is one "All Other Traffic" BT for each tier on which the limit is reached. Check if the expected BTs have been categorized as “All Other Traffic” by opening the dashboard for the “All Other Traffic - <tier name>” BT and clicking the View Traffic Details link. From here, you can see a list of entry points that were hit by incoming requests after the registration limit was maxed out. If needed, you can move transactions from the All Other Traffic bucket. 3. Do you need to revise your BT detection configuration? An important part of implementing AppDynamics is customizing your BT detection configuration so you can capture the most relevant transactions for your business. In addition to using automatic transaction discovery rules, you can use custom match rules  for more granular control. If customizations aren’t configured properly, you may not be capturing all of your expected BTs. For instructions on customizing detection rules, review the resources below: Transaction Detection Rules .NET Business Transaction Detection Java Business Transaction Detection Custom Exclude Rule Examples 4. Are the application agents reporting?   If your agents aren’t reporting business transaction data, there may be a connection issue between your agents and the Controller. Check your agent-to-Controller connections to see if this is the root cause for missing data: Verify agent-controller connection. For additional troubleshooting tips, visit: App Agent Not Reporting.
This flowchart describes the steps to confirm BT instrumentation when you are using a web framework. For a complete description of the BT Life Cycle, see: Business Transaction Discovery Life Cycle. ... See more...
This flowchart describes the steps to confirm BT instrumentation when you are using a web framework. For a complete description of the BT Life Cycle, see: Business Transaction Discovery Life Cycle. Supported Framework? First check our lists of supported environments to see if your framework is one that is discovered automatically by your agent: App Server Agents Supported Environments 1. Review the BCT Log Retrieve the Bytecode Transformer Log and look for entry point interceptors starting with entry.<your_framework> - for example, "entry.servlet" or "entry.struts". See Agent Log Files for details on requesting the agent logs. 2. Look for Entry Point Exceptions If you find evidence in the BCT log that the entry point was instrumented, look in the agent logs for entry point exceptions. See Request Agent Log Files for details on requesting agent logs.  If you find exceptions for your entry point, contact AppDynamics support for additional assistance.  If you do not see exceptions, review this article Business Transaction Discovery Life Cycle and use the steps at Phase 2 to verify registration with the controller. 3. Run Find Entry Points Use the procedure here Find Entry Points 4 Configure POJO or POCO Custom Match Rule For details see POJO Entry Points or POCO Entry Points. 5. Contact AppDynamics Support http://www.appdynamics.com/support/ 6. Verifying BT Registration If your entry point was instrumented and there were no related entry point exceptions, you can move to phase 2 - BT registration: Business Transaction Discovery Life Cycle.
To troubleshoot business transaction discovery it is helpful to have an understanding of the AppDynamics entity life cycle. A business transaction is a discrete piece of business application function... See more...
To troubleshoot business transaction discovery it is helpful to have an understanding of the AppDynamics entity life cycle. A business transaction is a discrete piece of business application functionality that is tracked from beginning to end, potentially across multiple tiers of processing, to measure the time it takes to complete the functionality. Phase 1: Business Transaction Discovery The discovery of a business transaction (BT) is accomplished by detecting where the business transaction begins and instrumenting its entry point. An entry point is the specific class and method (Java and .NET), operation or URI that kicks off the discrete bit of functionality that we want to measure. The exact entry points that are instrumented are agent dependent.  For details on dynamic language agents, see the supported environments pages for each agent in the product documentation: Supported Environments and Versions. Where does a BT begin? Where the core business logic is processed and executed. It might not necessarily be the first application component in the path of the request. A Java Servlet that handles a complex HTTP request is a good example of an entry point. In some cases the Servlet might merely forward the request to another component such as an EJB Bean that has the core business logic. In this case the EJB Bean is the right place to define the transaction as it is easier to differentiate one transaction from another based on the EJB Bean executing it. For PHP the entry point is typically a URI or a page callback name, MVC controller action, page template name, or page template. For Node.js the entry point is typically a method, operation or a URI.  Was BT Discovery Successful? Use the following steps: If you see BTs, but some that you expect to see are missing, first check that the BT limit was not exceeded. Was the BT limit reached? See All Other Traffic Business Transaction. If the BT limit was not reached or you do not find evidence of the business transaction in the traffic details of the overflow BT, go to the next step. Are you using a web framework? Yes, go to Confirming BT Instrumentation for Web Frameworks Are you running a background task? Yes, go to Confirming BT Instrumentation for Batch Frameworks If your entry point was not instrumented (see steps 3 and 4) and find entry points did not reveal a useful entry point, consider contacting AppDynamids Support at: http://www.appdynamics.com/support/. If you find evidence in the BCT log that the entry point was instrumented, check for entry point exceptions.        a. Look in the agent logs for entry point exceptions. For details on retrieving the logs, see Request Agent Log Files.        b. If you find exceptions for your entry point, contact support for additional assistance.        c. If you do not find any exceptions, move on to phase 2, registration. Phase 2: BT Registration When an entry point is successfully instrumented, it is assigned a name based on the rules for the entry point type or the custom naming rule (if applicable). Every BT is stored in the BT table in the controller database. The agent detects when the entry point is invoked and sends a BT registration request to the controller. Registration is when each BT is assigned an ID by the controller. After the ID is assigned, communication between the agent and controller uses the ID to identify exactly which BTs are being reported and stored in the controller database. You can review the BT registration log entries in the REST log. Use the following steps to confirm BT registration: 1. Generate and retrieve the REST log. For details on how to do this, see Request Agent Log Files. Look for registration and response details. In the following ViewCart BT request, you can see the entry point type is STRUTS_ACTION, the internal name is ViewCart.sendItems. At this point the id=0 and name=null. [AD Thread Pool-Global1|AD Thread Pool-Global1] 20 Nov 2012 09:49:44,491 DEBUG BusinessTransactionRegistry - REST - Sending Binary request - URL \[ /controller/instance/2/btregistration\] Payload \-****\**\**BT Registration Data*\******\* ... <AConfigObject ( AConfigObject ( com.singularity.ee.controller.api.dto.transactionmonitor.transactiondefinition.BusinessTransaction@8f000904 id = 0 name = null ) applicationId = 0 ) [ id=0, entryPointType=STRUTS_ACTION, internalName=ViewCart.sendItems, componentId=2, applicationComponentName=null, matchCriteria=com.singularity.ee.controller.api.dto.transactionmonitor.transactiondefinition.MatchCriteria@74cb7e2c, createdOn=null, background=false, configuration=null, createdNodeId=0, enabledForEUM=false, eumAutoEnablePossible=null]> In the following response you can see that the Controller assigned an id=80 and the name=ViewCart.sendItems. [AD Thread Pool-Global1] 20 Nov 2012 09:49:44,723 DEBUG BusinessTransactionRegistry - REST - Reading Binary payload - URL [ null] Size : [5376] Payload -********BT Registration Data******** ... <AConfigObject ( AConfigObject ( com.singularity.ee.controller.api.dto.transactionmonitor.transactiondefinition.BusinessTransaction@8f000904 id = 80 name = ViewCart.sendItems ) applicationId = 4 ) [ id=80, entryPointType=STRUTS_ACTION, internalName=ViewCart.sendItems, componentId=2, applicationComponentName=null, matchCriteria=com.singularity.ee.controller.api.dto.transactionmonitor.transactiondefinition.MatchCriteria@2ea28708, createdOn=null, background=false, configuration=null, createdNodeId=3, enabledForEUM=false, eumAutoEnablePossible=null]> 2. If you do not see a successful registration request and response, look in the Controller Logs for related exceptions. Phase 3 - BT metrics registration After successful BT registration, the agent keeps a BT map in memory with the ID, type, and name for each BT. The next time the entry point is hit, the agent is ready to report metrics. A similar process of request and response between the agent and controller occurs and the controller assigns each metric an ID. Use the following steps to confirm BT metric registration: Generate and retrieve the REST log. For details on how to do this, see Request Agent Log Files. Look for log entries showing metric registration. In the following request, you can see "BT:80", which we know from our review at phase 2 identifies the ViewCart.sendItems business transaction. The two metrics shown below represent Calls per Minute and Average Response Time for the ViewCart.sendItems BT. [AD Thread-Metric Reporter0] 20 Nov 2012 09:50:44,079 DEBUG MetricSender - Sending REST request - URL [ /controller/instance/2/metricregistration] XML payload -<request> ... <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:80|Component:2|Calls per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:80|Component:2|Average Response Time (ms)" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="INDIVIDUAL"/> In the following response you can see BT:80, identifying the ViewCart.sendItems BT. You can see that metric id=1815 was assigned to the Calls per Minute metric. A metric id=1813 was assigned to the Average Response Time metric. [AD Thread-Metric Reporter0] 20 Nov 2012 09:50:44,346 DEBUG MetricSender - REST Response Received -> <request> ... <metric id="1815" name="BTM|BTs|BT:80|Component:2|Calls per Minute"/> <metric id="1813" name="BTM|BTs|BT:80|Component:2|Average Response Time (ms)"/> 3.  If you do not see successful request and response for the metric, look in the Controller Logs for exceptions related to backend metric registration. Phase 4 - BT metrics reporting After successful, metric registration, BT metrics are reported to the controller every minute. Using the metric id, you can search the REST log for the metric upload. Use the following steps to confirm BT metric upload: 1. Generate and retrieve the REST log. For details on how to do this, see Request Agent Log Files. 2. Look for log entries showing metric reporting uploads similar to the following: [AD Thread-Metric Reporter0] 20 Nov 2012 09:50:44,361 DEBUG MetricSender - REST - Sending Binary request - URL [ /controller/instance/2/metrics] Payload -********Metric Data******** ... <metric id='1815', value[sum=32, count=1, min=32, max=32, current=32]> <metric id='1813', value[sum=8693, count=32, min=4, max=1614, current=53]> 3. If you do not see successful metric data uploads, look in the Controller Logs for related  exceptions. Phase 5 - BT metrics roll up App agents do not communicate with each other, they report metrics to the controller database. The controller aggregates the metrics from all nodes in each tier to present a unified overall picture in the Controller console. If you have verified that phases 1-4 of the BT discovery life cycle have been successful and you are not seeing the expected BT metrics, contact AppDynamics Support for assistance: http://www.appdynamics.com/support/.
Default JMX Metrics Example  Some MBeans exposed by various supported application servers are preconfigued as JMX Metrics that can be viewed in the JMX Metrics Browser. In this screen capture of... See more...
Default JMX Metrics Example  Some MBeans exposed by various supported application servers are preconfigued as JMX Metrics that can be viewed in the JMX Metrics Browser. In this screen capture of the JMX Metrics Browser, under Web Container Runtime, you can see the two metrics "Error Count" and "Request Count". Where did these come from? These metrics were configured from MBeans exposed by the Tomcat web container.  To see the configuration for these metrics, do the following: 1. Select Configure -> Instrumentation. 2. Select the application. 3. Select the JMX tab. In this "metric rule" named Tomcat_GlobalRequestProcessor", you can see where the Metric Path and the MBean matching criteria in the domain "Catalina" have been specified. This path in the metric browser was seen in the first screen capture. The next screen capture shows the two attributes of this MBean.
Question Our application uses JTurbo JDBC driver to connect to a MS SQL Database. We can not see the JDBC calls to the MS SQL Server database backend. How can we fix this? Answer Refer to this... See more...
Question Our application uses JTurbo JDBC driver to connect to a MS SQL Database. We can not see the JDBC calls to the MS SQL Server database backend. How can we fix this? Answer Refer to this article  Using Node Properties to Detect JDBC Backends and try using the following JDBC node properties and values: jdbc-connections=com.ashna.jturbo.driver.c jdbc-statements=com.ashna.jturbo.driver.w jdbc-prepared-statements=com.ashna.jturbo.driver.x jdbc-callable-statements=com.ashna.jturbo.driver.y Add all the above JDBC properties to the respective node and apply load to the application.
This article represents the latest collected set of wisdom from AppDynamics field engineers. For information on how to configure the Java Agent with TIBCO, see Configure the Java Agent for TIBCO... See more...
This article represents the latest collected set of wisdom from AppDynamics field engineers. For information on how to configure the Java Agent with TIBCO, see Configure the Java Agent for TIBCO BusinessWorks   Required Agent version BusinessWorks support requires version 3.9.6 or higher of the AppDynamics Java agent.    To get visibility of database calls AppDynamics provides agent node properties that can be used to detect and instrument JDBC backends that are not detected automatically.  The node properties are described here: Knowledge-Base: Using Node Properties to Detect JDBC Backends. You access the Node Dashboard to edit the value of these properties. Use these steps: Edit Registered Node Property. Tip: Separate multiple class names using a ',' (comma) as a separator in the agent node properties configuration. For Tibco,  use the following JDBC node property values:  jdbc-statements: value="tibcosoftwareinc.jdbc.base.BaseStatement,tibcosoftwareinc.jdbc.base.BaseCallableStatement,tibcosoftwareinc.jdbcx.base.BaseStatementWrapper,tibcosoftwareinc.jdbc.base.BasePreparedStatementPoolable" jdbc-connection: value="tibcosoftwareinc.jdbcx.base.BaseConnectionWrapper,tibcosoftwareinc.jdbc.base.BaseConnection jdbc-prepared-statements: value="tibcosoftwareinc.jdbc.base.BaseStatement,tibcosoftwareinc.jdbc.base.BaseCallableStatement,tibcosoftwareinc.jdbcx.base.BasePreparedStatementWrapper,tibcosoftwareinc.jdbc.base.BasePreparedStatementPoolable" jdbc-callable-statements: value="tibcosoftwareinc.jdbc.base.BaseCallableStatement,tibcosoftwareinc.jdbcx.base.BaseCallableStatementWrapper,tibcosoftwareinc.jdbc.base.BaseCallableStatementPoolable"/>  To track jobs through BusinessWorks The following configuration is provided out-of-the-box in AppDynamics release 3.9.7+ and 4.0+. You do  not need to change the app-agent-config.xml file unless your agent is 3.9.6 or earlier or you retained your existing, pre-4.0 app-agent-config.xml file after upgrading to 4.0.   <<<< This should be rare. For 3.9.6 or earlier agent, or if you retained your existing, pre-4.0 app-agent-config.xml file after upgrading to 4.0, you need to revise the configuration manually as shown here: 1. Locate and open for the app-agent-config.xml file for editing. 2. Make these changes in app-agent-config.xml:     a. Add the following to the  <fork-config> element: <!-- special config for tibco --> <job> <match-class type="matches-class"> <name filter-type="EQUALS" filter-value="com.tibco.pe.core.Job"/> </match-class> <match-method> <name filter-type="EQUALS" filter-value="k"/> </match-method> <name-config operation="" type="4"/> <retention-config type="1" operation="1.getTaskSize()"/> </job>     b. Uncomment this additional node property to allow detection of incoming JMS messages: <!-- uncomment the following to enable transaction correlation for jms .receive() call, default value is false --> <property name="enable-jms-receive-correlation" value="true"/>      c. Add the following entry to the <fork-config> element of app-agent-config.xml: <excludes filter-type="STARTSWITH" filter-value="com.tibco.plugin.share.jms.impl.JMSReceiver$SessionController”/> 3. Add the attached custom-interceptors.xml file to the <app-agent-install>/conf/ directory. To split SOAP/HTTP invocations by the SOAP action Many incoming SOAP actions that are sent via HTTP requests hit a single URL (for example, the URLs start with something like /BusinessServices/WebGateway) so you need a POJO split rule as shown in the two following screen shots. Match rule: Split rule: (see below for the values that are not showing completely in the screen shot). POJO Method Call values: Class Name = com.tibco.bw.service.binding.soap.http.SoapHttpTransportApplication Method Name = processMessage Method Call Chain = getTransportContext().getSoapAction() results in the following XML <custom-match-point-definition transaction-entry-point-type="SERVLET"> <name>get soap action</name> <custom-business-transaction-name>get soap action</custom-business-transaction-name> <background>false</background> <enabled>true</enabled> <match-rule> <servlet-rule> <enabled>true</enabled> <priority>50</priority> <uri filter-type="STARTSWITH" filter-value="/BusinessServices/WebGateway"/> <properties/> <generic-method-config> <class-name>com.tibco.bw.service.binding.soap.http.SoapHttpTransportApplication</class-name> <method-config> <name>processMessage</name> <param-length>1</param-length> <param-index>0</param-index> <param-getter>getTransportContext().getSoapAction()</param-getter> </method-config> </generic-method-config> </servlet-rule> </match-rule> </custom-match-point-definition>   If things don't seem to be working In this case, the TIBCO code is obfuscated.  You may be dealing with a different TIBCO BW release, and some of the configuration referencing TIBCO classes may need to change. Contact AppDynamics Support for additional assistance. Also see: Tibco ActiveMatrix BusinessWorks Service Engine Settings
For JBoss EAP AppDynamics supports the underlying JVM and framework versions. Our product documentation has specific installation instructions for JBoss (open-source versions) but those instructions ... See more...
For JBoss EAP AppDynamics supports the underlying JVM and framework versions. Our product documentation has specific installation instructions for JBoss (open-source versions) but those instructions do not necessarily work for JBoss EAP. JBoss EAP (Enterprise Application Platform) is Red Hat's branded and supported version of JBoss AS 7. JBoss AS 7 is already one of the more complex agent integrations, requiring modifications to two files in standalone mode: standalone.conf and standalone.sh. JBoss EAP complicates this integration by having a different layout which changes the location, names and content of these jar files. Documented Settings You can find settings in the public product documentation at JBoss and Wildfly Startup Settings for the following: Linux environment for JBoss EAP 6.11, EAP 6.2.0, and JBoss AS 7.0.x (standalone) RHEL JBoss EAP 6.x, JBoss AS 7.0.x, JBoss 8 (Domain Mode) JBoss 7.2 (standalone) Wildfly 8 (JBoss - the current steps for JBoss 7.x/EAP 6.x work for Wildfly 8. JBoss EAP 6.1 (Standalone) There is a JBoss EAP 6.1.0 bug that prevents our agent from working. This is fixed in JBoss EAP 6.1.1. JBoss EAP 6.1.0 bug reference (969530) under the 6.1.1 release notes https://access.redhat.com/site/documentation/en-US/JBoss_Enterprise_Application_Platform/6.1/html-single/6.1.1_Release_Notes/index.html#sect-changes. To get around this issue, consider an upgrade to JBoss EAP 6.1.1. The workaround at Step 2 under Quick Install here: JBoss and Wildfly Startup Settings may also work. Troubleshooting JBoss configuration The standard recipe for sorting out JBoss configuration is the following: 1. Don't add "-XX:-UseSplitVerifier" unless you need to. This is to workaround the case where your class files don't have a StackMap table such as for use with a debugger. http://stackoverflow.com/questions/15253173/how-safe-is-it-to-use-xx-usesplitverifier   2. JBoss EAP 6.10.0 is an OSGI server which changed classloading. It is now a modular classloader. The option -Djboss.modules.system.pkgs=org.jboss.byteman,com.singularity adds these classes to the OSGI container classpath: org.jboss.byteman    This is for debugging. See http://byteman.jboss.org/ You probably don't need this.   3. Setting the following logging options can be problematic. For example: org.jboss.logmanager in - Djboss.modules.system.pkgs  -Djava.util.logging.manager = org.jboss.logmanager.LogManager -Xbootclasspath /p:/var/jboss-eap-6.1.0/jboss-eap-6.1/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.4.0.Final-redhat-1.jar ... In theory these seems to be influenced by the application. If you need to set these, then you need to get the right modules associated in the bootclasspath so check the versions. They change. In practice it seems you can often get away without specifying them, saving a world of pain.   In JBoss EAP 6.1.0, this  seems to trigger a JBoss bug which cannot be worked around using the bootclasspath: https://bugzilla.redhat.com/show_bug.cgi?id=969530 - also referenced earlier in this article.  
Confirming Backend Discovery and Instrumentation This article describes the backend discovery life cycle. Understanding these details can help you debug instrumentation issues of backends that are ... See more...
Confirming Backend Discovery and Instrumentation This article describes the backend discovery life cycle. Understanding these details can help you debug instrumentation issues of backends that are not automatically discovered. The phases referred to in this article refer to the phases in the discovery life cycle. See AppDynamics Auto-Discovery Life Cycle. Typical examples of backends are databases instance and remote services, such as web services, JMS clients and message queues, HTTP backends and so on. Backends are born from exit points. An exit point call from an instrumented node to an uninstrumented node or other service results in backend discovery. The activity of a supported remote service or database is identified by its type and related properties. Each type of backend has a list of properties associated with it. The properties are a list of key-value pairs. For example, for a JDBC database, the following screen capture shows the properties that can be used to uniquely identify the JDBC database. Phase One: Was Backend Discovery Successful? Determine if the exit point was successfully instrumented: 1. Was the backend limit reached? If you see backends, but some are missing, check that the backend limit was not reached. There is a limit of 300 per application. If you have hit the backend limit, you may need to revise the backend discovery configuration. See All Other Traffic Backends. 2. If the backend limit was not reached, is your backend one that is discovered by default? To determine this, check the documentation for your app agent type. Use the docs version that corresponds to your agent version:  Latest: Supported Environments and Versions 3.9 Supported Environments and Versions  3. If your backend is not on the supported list, you can configure a custom exit point. See Configure Custom Exit Points. 4.  If you are using a supported backend and you are not seeing what you expect, find the BCT log (Byte Code Transformer log) and review the entries for exit point interceptors starting with the text exit.<your_framework>. For example, exit.jdbc for a JDBC database or exit.jms for a JMS service. See Agent Log Files for details on requesting the agent logs.       a. If you find evidence in the BCT log that the exit point was instrumented, check for exit point exceptions in the agent logs. If you find exceptions for your exit point, contact support for additional assistance. If you do not find any exceptions, move on to phase 2, backend registration.       b. If your exit point was not instrumented,  and you know the exit point that you want to instrument, configure a custom exit point. If you do not know the exit point to instrument, look at the Call Graph to find the method and configure the custom exit point. Phase Two: Backend Registration When a backend is successfully discovered, it is assigned a name based on the rules for the exit point type and any custom backend detection configuration (if applicable). Every backend is stored in the backend table in the controller database. The agent detects when the backend is hit and sends a registration request to the controller. Registration is the process where each object is assigned an ID by the controller. After the ID is assigned, the communication between the agent and controller uses the ID to identify exactly which objects are being reported and stored in the controller database. You can review the backend registration log entries in the REST Log. Confirm successful backend registration: 1. Generate and retrieve the REST log. For details on how to do this, see Request Agent Log Files. 2. Look for the backend registration request log entries. In the following request, you can see three backends were discovered and sent to the controller for registration.  <unregistered-backends> Size : 3 <UnresolvedBackendCallInfo [resolutionInfo=NodeResolutionInfo[exitPointType=JMS, properties=[Name:DESTINATION_TYPE, Value:QUEUE, Name:DESTINATION_NAME, Value:OrderQueue, Name:VENDOR, Value:Active MQ]], metaInfo=[], applicationId=0, createdOn=0, displayName=Active MQ-OrderQueue, visualizationProperties=null, applicationComponentNodeId=0, applicationComponentId=0]> <UnresolvedBackendCallInfo [resolutionInfo=NodeResolutionInfo[exitPointType=WEB_SERVICE, properties=[Name:SERVICE_NAME, Value:OrderService]], metaInfo=[], applicationId=0, createdOn=0, displayName=OrderService, visualizationProperties=null, applicationComponentNodeId=0, applicationComponentId=0]> <UnresolvedBackendCallInfo [resolutionInfo=NodeResolutionInfo[exitPointType=JDBC, properties=[Name:HOST, Value:LOCALHOST, Name:PORT, Value:3306, Name:SCHEMA, Value:APPDY, Name:MAJOR_VERSION, Value:5.5.16-log, Name:URL, Value:jdbc:mysql://localhost:3306/appdy, Name:VENDOR, Value:MySQL DB]], metaInfo=[], applicationId=0, createdOn=0, displayName=APPDY-MySQL DB, visualizationProperties=null, applicationComponentNodeId=0, applicationComponentId=0]> <unregistered-backends/> 3. Look for the corresponding backend registration responses.  In the following response you can see that the JMS exit point has an id=12 and is named  OrderQueue. ID 15 was assigned to the MySQL JDBC backend.  <resolved-backend-calls> <12>::<NodeResolutionInfo[exitPointType=JMS, properties=[Name:DESTINATION_NAME, Value:OrderQueue, Name:DESTINATION_TYPE, Value:QUEUE, Name:VENDOR, Value:Active MQ]]> <14>::<NodeResolutionInfo[exitPointType=WEB_SERVICE, properties=[Name:SERVICE_NAME, Value:OrderService]]> <15>::<NodeResolutionInfo[exitPointType=JDBC, properties=[Name:HOST, Value:LOCALHOST, Name:MAJOR_VERSION, Value:5.5.16-log, Name:PORT, Value:3306, Name:SCHEMA, Value:APPDY, Name:URL, Value:jdbc:mysql://localhost:3306/appdy, Name:VENDOR, Value:MySQL DB]]> ... <resolved-backend-calls/> 4. If you do not see successful registration request and response, look in the Controller Logs for exceptions related to backend registration. Note: If you exceed the backend limit, you see log messages similar to the following: AD Thread Pool-Global465] 23 Dec 2014 10:36:16,587 INFO BTOverflowCounter - AD Thread Pool-Global465] 23 Dec 2014 10:36:16,587 INFO BTOverflowCounter - DroppedBackend{key='https://rxservice.company.com/RxService/Account MgmtService', count=35, identifyingProperties={SERVICE_NAME=AccountMgmtService}, exitType=WEB_SERVICE} Phase Three: Metrics Registration After successful backend registration, the agent keeps a backend map in memory with the ID, type, and name for each backend. The next time the backend is hit, the agent is ready to report metrics. A similar process of request and response between the agent and controller occurs and the controller assigns each metric an ID. Was backend metric registration successful? There are three types of backend metrics: Business transaction (BT) metrics - aggregate metrics for the specific BT for the specified backend. Tier metrics - aggregate metrics across the tier for the specified backend. Backend metrics - overall aggregate metrics for the backend Use the following steps to confirm metric registration: 1. Generate and retrieve the REST log. For details on how to do this, see Request Agent Log Files. 2. Look for log entries showing metric registration. For example to find metrics related to the MySQL database registered in the previous example (ID=15), a        search of the REST log for the string "[UNRESOLVED][15]" results in the following types of log entries: <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:80|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Errors per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:81|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Average Response Time (ms)" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="INDIVIDUAL"/> <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:82|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Calls per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:80|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Average Response Time (ms)" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="INDIVIDUAL"/> <metric time-rollup-type="AVERAGE" name="BTM|Application Summary|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Average Response Time (ms)" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="INDIVIDUAL"/> <metric time-rollup-type="AVERAGE" name="BTM|Backends|Component:{[UNRESOLVED][15]}|Average Response Time (ms)" hole-fill-type="REGULAR_COUNTER" cluster-rollup-type="INDIVIDUAL"/> <metric time-rollup-type="AVERAGE" name="BTM|BTs|BT:80|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Calls per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|Application Summary|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Calls per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> <metric time-rollup-type="AVERAGE" name="BTM|Backends|Component:{[UNRESOLVED][15]}|Calls per Minute" hole-fill-type="RATE_COUNTER" cluster-rollup-type="COLLECTIVE"/> The metrics with names such as "BTM|BTs|BT:80|...", "BTM|BTs|BT:81|...," and "BTM|BTs|BT:82|..." are BT backend metrics. You may recognize the BT ID numbers, 80,81, and 82. The metrics with names such as "BTM|Application Summary..." are tier level metrics for this backend. The metrics with names such as "BTM|Backends|... " are aggregate backend metrics. 3.  Look for the corresponding metric registration response entries. In the following response you can see the IDs assigned to the metrics. For example, the first line in the example shows ID=1925 is assigned to the "Errors per Minute" metric for BT 80 on this MySQL backend (UNRESOLVED 15). <metric id="1925" name="BTM|BTs|BT:80|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Errors per Minute"/> <metric id="1936" name="BTM|BTs|BT:81|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Average Response Time (ms)"/> <metric id="1959" name="BTM|BTs|BT:82|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Calls per Minute"/> <metric id="1967" name="BTM|BTs|BT:80|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Average Response Time (ms)"/> <metric id="1935" name="BTM|Application Summary|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Average Response Time (ms)"/> <metric id="2015" name="BTM|Application Summary|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Calls per Minute"/> <metric id="1937" name="BTM|Backends|Component:{[UNRESOLVED][15]}|Average Response Time (ms)"/>... <metric id="1991" name="BTM|BTs|BT:80|Component:2|Exit Call:JDBC|To:{[UNRESOLVED][15]}|Calls per Minute"/> <metric id="2017" name="BTM|Backends|Component:{[UNRESOLVED][15]}|Calls per Minute"/> 4. If you do not see successful request and response for the metric, look in the Controller Logs for exceptions related to backend metric registration. Phase Four: Backend Metric Reporting After successful metric registration, metrics are reported to the controller every minute. Using the metric id, you can search the REST log for the metric upload. Was backend metric reporting successful? Use the following steps to confirm backend metric upload: 1. Generate and retrieve the REST log. For details on how to do this, see Request Agent Log Files. 2. Look for log entries showing metric reporting uploads similar to the following: <metric id='1936', value[sum=177, count=499, min=0, max=7, current=1]> <metric id='1959', value[sum=160, count=1, min=160, max=160, current=160]> <metric id='1967', value[sum=157, count=518, min=0, max=4, current=0]> <metric id='1935', value[sum=611, count=688, min=0, max=79, current=1]> <metric id='2015', value[sum=1353, count=1, min=1353, max=1353, current=1353]> <metric id='1937', value[sum=285, count=1353, min=0, max=2, current=0]> <metric id='1991', value[sum=136, count=1, min=136, max=136, current=136]> <metric id='2017', value[sum=1144, count=1, min=1144, max=1144, current=1144]> 3. If you do not see successful backend metric data uploads, look in the Controller Logs for related exceptions. 
AppDynamics automatically discovers various entities that make up your web applications, such as app servers, databases, and remote services. Other activity in your application, such as errors, excep... See more...
AppDynamics automatically discovers various entities that make up your web applications, such as app servers, databases, and remote services. Other activity in your application, such as errors, exceptions, async activity, and business transactions are also tracked as AppDynamics entities. You can learn to do some of your own debugging when things don't seem to be showing you exactly what you want to see if you understand the AppDynamics entity life cycle. The discovery life cycle is essentially the same for all entities and involves several discrete steps that can be identified by log entries. The discovery life cycle contains the following four steps for all entities: Discovery The application name, tier name, and node name are passed to the agent as part of the startup process. When the agent starts up, these entities are "discovered". The discovery of a Business Transaction (BT) is about detecting where the business transaction begins and instrumenting its entry point. An exit call from an instrumented node to an uninstrumented node or other service results in backend discovery of database, message queues, web services, and other remote service layers. Error detection occurs when the agent detects an error or exception in the application. Related documentation:  AppDynamics Concepts Monitor Business Transactions Monitor Errors and Exceptions    Registration Registration is the process where each entity is assigned a unique ID by the controller. The agent sends a registration request to the controller to register the discovered entity. The controller assigns a unique id. This ID is then used to identify this entity in agent-controller communication and in the controller database. Metric Registration As load is applied to the application, the agent starts collection metrics for each entity. Each metric is reported to the controller. The controller registers the metric and assigns an ID. The ID is used in agent-controller communication. Metric Reporting After successful metric registration, the agent periodically reports metrics to the controller.  Metric Rollup For business transactions, tiers, and nodes, there is an additional step that occurs at the controller itself. That step is metrics rollup. App agents do not communicate with each other, they report metrics to the controller. The controller aggregates the metrics from all agents/nodes to present a unified overall picture in the Controller UI. Verify Each Step in the Life Cycle  Using the agent logs it is possible to verify whether each step in the life cycle has occurred successfully. 
Contents: Who would use this workflow? How to know which POJO to use? Implementation Limitations Who would use this workflow?  Note: This workflow has largely been replaced by the Li... See more...
Contents: Who would use this workflow? How to know which POJO to use? Implementation Limitations Who would use this workflow?  Note: This workflow has largely been replaced by the Live Mode feature. If you don't see expected activity in your application, you might be missing an entry point at the start of the activity. Use the find-entry-points node property to configure missing entry points for any Business Transactions (BTs) that aren't detected out-of-the-box (OOTB). This property enables additional logging about the call stack of the app's executing code. In many cases, the OOTB configuration instruments entry points so this workflow isn't usually necessary. You would only use these instructions if your code is built within a framework that isn't supported by the out-of-the-box configuration. However, we recommend using the Live Mode feature instead. How do I know which POJO to instrument? Three rules to keep in mind: Select the POJO at the earliest point possible in the call stack. Select a POJO that finishes. You are instrumenting a POJO to measure the time the business transaction takes to execute, so you must select a POJO that finishes. Select a POJO that results in a meaningful name. The entry point method determines the name of the BT. NOTE: Do not pick a method that never ends, such as a run () method that waits on a user request! See: POJO Entry Points. Implementation: Navigate to node-level agent configuration and set the find-entry-points property to true. For more information on how to edit a node-level property: Edit a Registered Node Property Apply the property either tier-level or node-level. Note: Do not turn on this property tier-level within a production environment. The find-entry-points property is meant to be used under pre-prod and non-prod environments. Once the property is applied, apply sufficient load on the application and start collecting agent logs from the Controller UI, or view them from agent installation folder. Debug logging is not needed for this property. A complete call stack from the instrumentor to the top of the thread will dump to the   BusinessTransactions<X>.log file. Example: BusinessTransactions.2017_11_27__12_54_08.0.log The BT log mixes the output of find-entry-points with the "normal" BT discovery logging. To distinguish between the two types of call stacks, identify the text after the INFO log level keyword. Entire stack trace with eligible entry point prioritized: Based on this output, the user can find eligible candidates and create custom POJO entry points.   One can create a custom POJO entry point on class com.xyz.abc.domain.pqr.Health and method run. In the above example, 0th is considered as most eligible entry priority. Based on the output, a custom entry point rule can be created on any type (ex: Servlet, EJB, Spring or POJO). More information: Custom Match Rules. Note: Setting the property value to true causes verbose logging to be output until the value is switched to false, so it is unwise to leave this property value set to true. Limitations If the application has no known exit calls, such as no HTTP, web services, JDBC, etc, then no call stacks are logged when you run find entry points. To work around this limitation, set a custom exit point to force the logging of the associated call stack, then run find entry points again. The log entry at the top of the call stack should be your custom exit point. This works best when you know a method in the business functionality or user request that you are trying to measure. See Configure Custom Exit Points. Published on 2/13/2015 Updated 2/11/19
After installing AppDynamics Database Agent, if you see error messages similar to the following code the issue may be with OS monitoring for your Database collector. 15:33:48,718 ERROR [Agent-Sche... See more...
After installing AppDynamics Database Agent, if you see error messages similar to the following code the issue may be with OS monitoring for your Database collector. 15:33:48,718 ERROR [Agent-Scheduler-1] AServerCollector:199 - Error collecting hardware metrics for server 'kpi' com.singularity.ee.agent.dbagent.collector.server.connection.ServerConnectionException: org.jinterop.dcom.common.JIException: Message not found for errorCode: 0xC0000001 at com.singularity.ee.agent.dbagent.collector.server.connection.WMIConnection.<init>(WMIConnection.java:66) The message indicates that the WMI permissions were not properly set to collect hardware metrics from the OS. Validate your WMI security permissions as described in the documentation WMI Permissions and Security.
Users who are troubleshooting scenarios where the AppDynamics *Standalone Machine Agent is not reporting metrics as expected will find causes and solutions here. The following solutions apply to ve... See more...
Users who are troubleshooting scenarios where the AppDynamics *Standalone Machine Agent is not reporting metrics as expected will find causes and solutions here. The following solutions apply to versions 4.4 and higher. Contents: Installation issues Host ID for App Server Agent and Machine Agent do not match Difficulty with machine hostname resolution Machine Agent reporting all metrics as zero Problem: Installation issues If both an App Server Agent and the Machine Agent are unzipped into the same directory, important files will be overwritten, such as the  log4j.xml  file. Solution: When unzipping the App Agent or Machine Agent zip files, make sure to use different directories. Problem: Host ID for App Server Agent and Machine Agent do not match  Both the App Server Agent and the Machine Agent use the Java API to get the host ID. The results from the API can be inconsistent and the same JVM can sometimes return a different value for the same machine each time the Machine Agent is restarted.  When the host ID is then registered with the Controller the app server agent and the machine agent can be assigned different host IDs even though they are running on the same machine. Solution: Reset the hostname for the Machine Agent by running the agent with the -Dappdynamics.agent.uniqueHostId  JVM parameter. Set the host ID to be the same as the one the App Server Agent is using.  Problem: Difficulty with machine hostname resolution If the following error is seen in the Machine Agent log file, the cause is hostname resolution, which affects Machine Agent startup and registration. .ERROR XMLConfigManager - Error in Default Host Identifier Resolver resolving host name java.net.UnknownHostException: log-aggregate01: log-aggregate01   Solution: 1. Verify a valid hostname in the  /etc/hosts  file. Example: localhost    log-aggregate01 127.0.0.1         log-aggregate01 2. Save the changes. 3. Restart the Machine Agent.   Problem: Machine Agent reporting all metrics as zero You have verified in the Machine Agent log that the Machine Agent is only collecting zero-values.    Solution: 1. Change permissions on the Machine Agent installation folder for the user ID that the Machine Agent was started under  chmod -R 777 <machine-agent-install > . 2. Restart the Machine Agent and verify that the zero values are no longer being reported. 3. If restarting does not help, disable Sigar Hardware monitoring located in the  JavaHardwareMonitoring  directory. In the  monitor.xml  file (located at:  <machine-agent-install>/monitors/JavaHardwareMonitor/* ), change the enabled property to false. Example: <monitor> <name>SigarHardwareMonitor</name>  <type>managed</type>  <enabled>true</enabled> ---> change this to false –> <enabled>false</enabled> 4. Enable OS-specific hardware monitoring found in the  HardwareMonitoring  directory.  In  monitor.xml  file (location:  <machine-agent-install>/monitors/HardwareMonitor/* )  <monitor>  <name>HardwareMonitor</name>  <type>managed</type>  <enabled>false</enabled> ---> change this to true –> <enabled>true</enabled> 5. Restart the Machine Agent and verify that the zero values are no longer being reported.     Last Content Update 3/28/19   __________________ *As of 2020, the term "Standalone Machine Agent" has been discontinued in favor of simply "Machine Agent.
Problem: App Agent log is empty, or nothing is reported to the Controller. Common Solutions: If your agent is behind a network firewall or load balancer, open ports to enable communication be... See more...
Problem: App Agent log is empty, or nothing is reported to the Controller. Common Solutions: If your agent is behind a network firewall or load balancer, open ports to enable communication between the agent and the Controller. Avoid installing an App Agent into a directory used by the application server, such as a Tomcat directory. Always install the Java Agent to a directory of its own, such as \usr\local\agentsetup\appserveragent . The AppDynamics folder is installed by the same user who owns the application process. The AppDynamics folder must have read and write permissions. Check for the following error in the application logs while installing the Java Agent. This error means that the Java Agent folder is corrupt. Install a new Java Agent folder. Error opening zip file or JAR manifest missing : \usr\local\agentsetup\appserveragent\javaagent.jar Is the runtime directory writeable by the Java Agent? See Controller Port Settings. Check for network connectivity issues. If the agent is not able to connect to the Controller, the agent will disable itself. When the connection is available it will re-register again. Validate the application name, tier names, and node names. These are mandatory configuration parameters for an agent. Examine logs for errors such as the following:   Additional Information: Troubleshooting Agent Issues - Reporting and Connectivity How to resolve Java Agents not reporting Documentation: Troubleshooting Java Agent Issues Documentation: Connect the Controller and Agents Documentation: Dynamic Language Agent Proxy
Content revised  8/1/18 You may want to configure Error Detection to ignore known exceptions. For example, you may see a recurring exception thrown from a framework you are using. If the exception ... See more...
Content revised  8/1/18 You may want to configure Error Detection to ignore known exceptions. For example, you may see a recurring exception thrown from a framework you are using. If the exception is well-known and insignificant, you might want to eliminate it from the error list. Contents What you need to know about each error Example | Ignoring Error Messages Example | Excluding Exceptions Example | Ignoring Exceptions Related Resources What you need to know about each error The first thing you need to know for each message is whether AppDynamics is detecting it as a thrown exception or a message written to a log file with severity ERROR. If it is an exception, the snapshot's Error Details tab should have a stack trace. If you only see the error message, it's a logger. EXAMPLE | Logged error message: You can dispose of it by ignoring logged messages such as: Example | Ignoring Error Messages This is the error message: com.appdynamicspilot.persistence.ItemPersistence.com.appdynamicspilot.persistence. ItemPersistence : Critical transaction Error, rolling back changes. Order execution aborted. Below is the configuration for ignoring the error message: Example | Excluding Exceptions The next two examples are thrown exceptions, both from the same place: Response-xslFile :null; IsActivationCompleteSuccess:false; IsValidated :false; OrderType :QUERY; AccountNumber :null; CustomerQualification :null; CapErrorCode :00602; capErrorType :1; CapErrorCategory :CDE; CapErrorDesc :Account_Data_Discrepency; OperationMode :null; OperationStatus :null; ActivationDate :null SUBMIT_ACTIVATION_DETAILS : Transaction Id:xx-23341105-1 - Controller:handleRequest() > An Exception is caught Dependent object contains ... [details deleted] CarrierErrorCode :null; The exception message is :com.company.bb.cap.exceptions.CarrierBusinessRulesException: Controller:handleRequest(): Transaction failure response from Carrier. You can exclude these by specifying  com.company.bb.cap.exceptions.CarrierBusinessRulesException as the class and ignoring particular values of CarrierErrorCode (which are buried in the full message).   Is the exception nested? If it is a true exception, please note whether it's nested inside another exception. You'll need to specify the exact sequence in the Error Detection config. It will look like this: java.net.WebException : com.company.bb.cap.exceptions.CarrierBusinessRulesException: Controller:handleRequest(): Transaction failure response from Carrier .   Check the stack trace Pay close attention to what is in the stack trace. For example, a stack trace similar to the following might show up in the Error Details section of a snapshot: java.lang.reflect.InvocationTargetException:com.company.eppi.client.exceptions. ClientException: java.lang.reflect.InvocationTargetException.null at sun.reflect.GeneratedMethodAccessor1249.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) ... <details deleted> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: com.comodo.epki.client.exceptions.ClientException : com.comodo.epki.client.exceptions.ClientException: Not authorized agent at com.comodo.epki.extra.agent.ws.CcmExtraAgentServer.authorizeAgent (CcmExtraAgentServer.java:210) at com.comodo.epki.extra.agent.ws.CcmExtraAgentServer.getExtraCommand (CcmExtraAgentServer.java:70) Don't create the exclusion on the "caused by" exception. Instead, use the first one in the stack trace. In the example above, you would set the Fully Qualified Class Names for Exceptions field as java.lang.reflect.InvocationTargetException:com.company.eppi.client.exceptions.ClientException . You would not put it on com.company.epki.client.exceptions.ClientException . The configuration for this example would like the screenshot below:   Ignoring Exceptions Example This is the exception: org.springframework.transaction.CannotCreateTransactionException:org.eclipse.persistence. exceptions.DatabaseException:org.apache.tomcat.dbcp.dbcp.SQLNestedException:java.util. NoSuchElementException   Below is the configuration to ignore the error message: Related Documentation For more detailed information on error detection and specifically how to ignore exceptions and log messages, see: Ignore Exceptions and Log Messages as Error Indicators.
Sometimes errors may appear in an appserver log, but not in AppDynamics. Or you may run your own specific test suites and see that known errors are not being detected by AppDynamics. Here are some th... See more...
Sometimes errors may appear in an appserver log, but not in AppDynamics. Or you may run your own specific test suites and see that known errors are not being detected by AppDynamics. Here are some things to check. 1. Confirm Logging Framework is Supported 2. Confirm Error Limits Were Not Hit 3. Confirm Configuration for Ignored Exceptions, Errors, and Loggers 4. Missing HTTP Error Codes 1. Confirm Logging Framework is Supported Your appserver may be using an unsupported logging framework. AppDynamics App Agent for Java supports the following logging frameworks: Log4j 2 java.util.logging New in 4.0 Simple Logging Facade for Java (SLF4J) New in 4.0 Logback Also see Java Supported Environments for the latest support. AppDynamics App Agent for .NET supports the following logging frameworks: Log4Net NLog Also see .NET Supported Environments Scope of Support In version 4.0, support was added for SLF4J, Logback, which  implements SLF4J under the covers, and for Log4j 2. The support extends to the following features of these logging libraries: SLF4J, Logback We support instrumenting classes that implement the slf4j interface. Logback uses slf4j natively, so we support logback also.   Supported Methods: Logger.error(String) Logger.error(Marker, String) Logger.error(String, Throwable) Logger.error(Marker, String, Throwable) We do not support SLF4J error passed objects, for example, error(java.lang.String,%20java.lang.Object...)    Log4j 2.0  We instrument out of the box anything that implements the log4j2 Logger interface. Specifically, we support: error(Marker marker, Message msg) error(Marker marker, Object message) error(Marker marker, String message) error(Message msg) error(Object message) error(String message) error(Marker marker, Message msg, Throwable t) error(Marker marker, Object message, Throwable t) error(Marker marker, String message, Throwable t) error(Message msg, Throwable t) error(Object message, Throwable t) error(String message, Throwable t) Also, fatal variants of all of the above are supported.  Notice that we don't support logger.logMessage(), log(), or any calls with Object ... params (meaning a Object[] params). We don't support the log() and logMessage() from ExtendedLogger. For additional logging support The solution is to configure a custom logger. See Configure a Custom Logger. 2. Confirm Error Limits Were Not Hit Agent Error Limit There is an agent metric limit of 5000 metrics that can be registered per Node and an agent limit of 500 ADDs (Application Diagnostic Data - this includes async threads, errors and exception registration, snapshots and so on). If this limit is reached and the Agent attempts to create metrics beyond this threshold, you see AGENT_METRIC_REG_LIMIT_REACHED alert in the event list. You can increase this default limit but that might cause an increase in overhead. Sometimes hitting this limit can be indicative of misconfigurations in your application. Hitting this limit and a similar limit in the Controller can indicate that you have hit the business transaction or backend limits and you may need to change the default discovery rules. What is a metric? A metric is an identifier used to uniquely identify a particular statistic. For example: Application Infrastructure Performance|Author|JVM|Memory|Heap|Committed (MB) Application Infrastructure Performance|Author|JVM|Memory|Heap|Used % Application Infrastructure Performance|Author|JVM|Process CPU Usage % All of the above are individual metrics registered from the node, against which the corresponding statistics data is collected and reported to the controller. At any particular point in time, the metric name remains the same but the value changes and that value is captured and reported.  This particular concept of a metric is internal to AppDynamics, however, it is helpful to understand how it works because of the self-imposed limits on the number of metrics that can be discovered. The limits help to minimize the AppDynamics footprint and overhead impact to an application. One limit is the maximum number of metrics that the agent creates. Once the limit is reached the agent does not create new metrics. Q: What is the impact of exceeding this 5000 limit? A. This limit is per agent. Once the limit is reached no new metrics are created, therefore no new activity is tracked. If you have more endpoints discovered those are not tracked. Q: If that is true then does restarting the agent from the console reset this limit and hopefully get new endpoints monitored while perhaps not picking up some old defunct ones that were working towards the 5000 limit? A: Once the metric is registered, it is present always for that agent whether or not load is present on that metric or not. For example, once metrics corresponding to an HTTP backend are registered, it doesn't matter whether there are calls or not to that backend, those metrics are always counted against the limit. In a case such as this, you could increase the maximum metric limit or you can delete the backends that are not being used to free up those metrics. You may also need to revise your backend configuration to avoid registering so many backends. Once you increase the limit (or free up metrics by deleting unneeded backends/components), it is not guaranteed that the new end points will be visible because it is possible that there are other statistics which will be detected first and use the added metrics capacity. For example, if there are async calls in the application, but the agent was not able to create metrics for them due to the limit being reached, once there is metric capacity, those async-call-related metrics might be created first before the new endpoints are detected. Solution: Revise Configuration Verify that you are not exceeding other limits such as backend limits and BT limits. Hitting the metric limit can be a warning sign of a configuration problem specific to your environment. For example, HTTP backends are discovered automatically using Port and Host properties. If the configuration were changed to use the entire URL, your might rapidly  exceed the backend limits and cause the metric limit to be hit, when the real problem is the HTTP backend configuration. Solution: Increase Limit The limit of 5000 suffices much of the time, but if truly needed it can be increased if you think there are calls missing or some functionality is not being captured by the agent.  On average, agents register 800 metrics across applications. The lower end is 300 and some applications produce 1500 metrics per agent. If agents need more than 5000 metrics, something else is often wrong and raising the limit rather obscures the underlying problem. To increase the limit see Metrics Limits documentation. NOTE: Before increasing the metric limit, be sure you have verified no other limits are being hit. Solution: Modify Default Error Detection Rules If there are errors or exceptions that are well-known and don't need to be monitoried, you can exclude them from detection and free up metric capacity. Review the documentation here: Configure Exceptions and Log Messages to Ignore. Controller Metric Limit There is a similar metric limit at the controller level. When this limit is reached, the controller issues the CONTROLLER_ERROR_ADD_REG_LIMIT_REACHED event. Solutions The recommended solution is to fine-tune the default error detection rules,  for example exclude the ones you're not interested in. Review the documentation here: Configure Exceptions and Log Messages to Ignore. Increase the default limit to 4000, for example:        a. Login to the admin page http://<controller ip>:<port>/controller/admin.html       b. Enter root password (default value is changeme)       c. Change the value of the property 'error.registration.limit' accordingly (see attached screenshot) Note: Increasing the limit incurs additional overhead so be sure to verify that you need to monitor all the discovered errors and exceptions. 3. Confirm Configuration for Ignored Exceptions, Errors, and Loggers If exclude rules are misconfigured, exceptions might be missed. Review the error detection configuration in your application. From Controller UI -> select Configure -> Instrumentation -> select Error Detection tab. For more details, see: Monitor Errors and Exceptions   4. Missing HTTP Error Codes AppDynamics reports error codes when the sendError method is used to report the error code. However, for some implementations of HttpServletResponse some HTTP errors are sent using setStatus. In this case, pset the capture-set-status (Java agent only) node property to true to capture these HTTP errors. For more details see the node property reference documentation: App Agent Node Properties Reference.