Problem: Discovery of the Events Service node using Enterprise Console (EC) fails. Upon checking the EC logs, we see the below error: Events Service discovery failed.  Failure occurred at stage: Enable transient allocations on masters  Error message:  Task failed: Enable transient allocation in the api store node  Analysis: This is a very generic error that occurs if the Events Service is failing on its respective node. To get the actual cause of issue, we need to check the events-service-api-store.log . In the Events Service log for this instance, we see the below error: [2018-08-07T07:08:42,738+02:00]  [ERROR]  [ProcessQueue0]  [c.a.a.p.e.e.ElasticSearchExecutor]  Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000004e32c0000, 12294815744, 0) failed; error='Cannot allocate memory' (errno=12) The system did not have enough memory to allocate the process. Hence, the start of the process failed so the discovery process failed, as well. Solution: Ensure that there is enough space available for the Events Service and elastic search process combined so that the process can start and the discovery process will complete successfully. ... View more

Question  Can I enable multi-factor authentication for AppDynamics mobile applications? Answer Currently, there is no way to configure multi factor authentication for an AppDynamics mobile application. We only have direct username/password-based or SAML login for the mobile app.  ... View more

Problem: Analytics data being collected is a lot less than the actual load on the application and you are seeing repetitive messages like the one below in the Analytics Agent log: [2018-05-21T11:24:38,137+01:00]  [WARN ]  [pipeline-thread-12 (bt-publish-4)]  [c.a.a.a.p.e.EventServicePublishStage]  The message at index [0] failed due to the following permanent error: [Failed [failed to parse [segments.userData.xxxx]] with id [484xxx-63xx-4xxx-axxx-47d08xxxxx] index [customer1_xxxxxxx___biz_txn_v1___2018-05-19_16-38-20] type [customer1_xxxxx___biz_txn_v1]] Analysis: The same MIDC is defined for multiple applications. We observed that the field that is registered with the Events Service has a different type than the field that is being collected for the current application. The field type can be retrieved using the below query: curl -XGET http://localhost:9200/<global_account_name>___biz_txn_v1*/_mapping?pretty=true If same field name is being used across multiple MIDC, the Events Service will use the first registered field type. If the field type for a new application is different from the registered one, the Analytics Agent will not parse the record and will discard it.  It is not discarding all the requests because the MIDC field is not being collected for all events. Hence, the events that do not have this field value are parsed properly and published to the Events Service. For this reason, you will see less events data compared to actual load on the application. Solution: You can approach this in two ways: 1. You can change the type of data being returned by the data collector at the application level to the type that is registered with Events Service. 2. You can change the field name in the MIDC that is defined for the current application so that it registers with the Events Service with its own type under a new field. ... View more

Problem: EUM browser data is not correlating to the backend application, even when the correlation cookie is coming as a part of the response from the application server. Analysis: We see that the cookie information is present in the response and there is no httpOnly flag tagged with the cookie, but the same data is not being sent in the beacons. The guid we see in the beacon is self-generated and not the one sent by the application server. This is causing the correlation to break. We can see that there are resource files in the page like .png that also have the correlation cookies. These correlation cookies on resource files cause the actual page correlation to break. Solution: The transactions serving the resource files should not be enabled for correlation. Identify the transactions that serves the resource file on the page and disable EUM correlation for those transactions. ... View more

Question  Why do I need to run the eum-schema update after importing the MySQL eum_db dump from the Controller MySQL to the EUM MySQL database? Answer Starting 4.4, EUM has its own MySQL database in production mode deployment. If we are upgrading from a lower version to 4.4.x, we need to take a dump from the Controller MySQL for the  eum_db schema and import the dump in the EUM server's MySQL database. In 4.4.x, the schema of the EUM server has additional changes from its previous version, which are required for the EUM processor to function. When we are importing the schema level dump, it drops the current eum_db schema in the EUM database and then restores the older version of the schema from dump. To bring it back to a compatible state for the 4.4.x version, we need to update the schema using the eum-schema update  command. ... View more

I’m upgrading EUM from a version lower than 4.4. Why does the upgrade fail during “schema update”? Problem EUM upgrade from a version lower than 4.4 fails in the "schema update" step with the below error: Unexpected error running Liquibase: Error executing SQL CREATE TABLE eum_db.crash_alert_events (hash_key VARCHAR(128) NOT NULL, range_key VARCHAR(128) NOT NULL, application_key VARCHAR(128) NOT NULL, alert_minute BIGINT(19) NOT NULL, group_id VARCHAR(128) NOT NULL, threshold INT(19) NOT NULL): Table 'crash_alert_events' already exists Analysis In versions prior to v4.4, the EUM database schema uses Controller MySQL. Starting with v4.4, EUM has its own database for a production mode. So, for a production mode installation, you must export the schema from the Controller database into the EUM MySQL database.  The EUM schema in v4.4 has some additional tables. When we import the dump using the documented process, it drops the eum_db schema in MySQL of EUM and then imports the dump from the previous version. The update-schema then creates the tables that were missing in the previous schema version. If we are not using the recommended command to take the export, the tables that are different from the dump are not dropped during the import. Therefore, when we run the eum-schema update  command, it fails with the above error. For example, we can take the database export using the below command as well: ./mysqldump -u root --password --protocol=TCP -P 3388 eum_db --add-drop-database > eum_db.sql However, when we use this dump to import, it does not have drop syntax for the  eum_db schema , which poses a problem during the update-schema command.  Solution To avoid any anomalies during the upgrade process, use the specified commands for taking the dump, as below: ./mysqldump -u root --password --protocol=TCP -P [Controller-MySQL-Port] --databases eum_db --add-drop-database > eum_db.sql This will ensure that the eum_db schema is cleared before importing the dump, and that the eum-schema update  will run without issues. ... View more

Problem: Values for EURT show considerable differences between the Pages and Ajax Request view and the Analyze view.  Analysis: By design, the EUM does not include outlier requests. However, the outlier requests are sent to the Events Service. If there are more outlier requests, the EURT in Analyze can get spiked.  Solution: Your outlier threshold is five times the stall threshold. If you do not want to see outliers in the Analyze view, you can add a filter to the displayed results limiting the value to filter out the outlier requests. ... View more

Question Can we change background color of the analytics widget? Answer The ability to change the background colour for analytics widget is not present in the Controller UIversion 4.3.x or earlier. However, you can change the color starting 4.4.x under Widget Properties. ... View more

Question Can I change the default path of custom metrics published via API to display them under the analytics metric hierarchy or the mobile metrics hierarchy ? Answer By default, custom metrics have their own hierarchy for display in the Metric Browser by default. We cannot change this to have them displayed under any other metric hierarchy view.  ... View more

Problem: The Controller is not able to communicate to EUM server over https. We see the error below in the Controller server log: WARNING|glassfish 4.1|com.singularity.ee.controller.beans.eumcloud.EUMCloudManagerBean|_ThreadID=172;_ThreadName=http-listener-2(14);_TimeMillis=1520953028920;_LevelValue=900;|Unable to fetch/update information for EUM account with name: VodafoneUK-a0Q3400000EpfCOEAZ com.appdynamics.eum.rest.client.exception.TransportException: Communication failure with service (https://EUM_HOST_IP:7002/v2/account/eumAccountName/license): javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative names present.|#] Analysis: Sometimes in Controller configuration for the EUM end point, we provide the IP address and not the domain name. Hence, the certificate is also issued accordingly. For example, if the EUM end point is https://10.10.10.10:7002 then the CN for the certificate will be 10.10.10.10. In such cases, if the certificate is issued without SAN (subject alternative name), then the certificate becomes invalid as it is mandatory to have SAN in case the URL contains the IP address and not the domain name. If the SAN is absent in the certificate, the ssl handshake will fail with the error above. Solution: If you are using the IP address in the EUM URL, you need to generate the certificate with SAN. Below is the sample keytool command: keytool -certreq -file test.csr -keystore test.jks -alias testAlias -ext SAN=dns:test.example.com ... View more

Question Does the node property  eum-header-beforeendofheadtag apply to all type of application agents? Answer The node property  eum-header-beforeendofheadtag enables the application agent to write the automatic injection code before the end of the <head> tag to avoid any UI rendering conflict. Conflicts can occur because the JavaScript Agent injects code before other components within the head tag. This property is only recognized by the .NET Agent. This is not applicable to any other app agent instrumentation. ... View more

Question  How do I add custom user data via automatic injection? Answer 1. Navigate to the Browser App that is linked to the APM app and click on configuration > Configure and Download JavaScript Agent. 2. Expand the advanced section. In the custom configuration box, add the following: window['adrum-config'] = {  userEventInfo: {    "PageView": function(context) {      return{        userData: {          page_title: "RandomGeneratedTest"        }      }    }  } }; window['adrum-start-time'] = new Date().getTime(); (function(config){    config.appKey = 'BD-AAB-XYZ-XYZ';    config.adrumExtUrlHttp = 'http://cdn.appdynamics.com';    config.adrumExtUrlHttps = 'https://cdn.appdynamics.com';    config.beaconUrlHttp = 'http://col.eum-appdynamics.com';    config.beaconUrlHttps = 'https://col.eum-appdynamics.com';    config.xd = {enable : false}; })(window['adrum-config'] || (window['adrum-config'] = {})); 3. Click on Save Config & Generate HTML Snippet. 4. Verify in the page source if the code is being successfully injected: <html> <head> <script type='text/javascript'> //<![CDATA[ window['adrum-start-time'] = new Date().getTime(); ;(function(config){ config.appKey = 'BD-AAC-XYZ-XYZ'; config.beaconUrlHttp = 'http://col.eum-appdynamics.com'; config.beaconUrlHttps = 'https://col.eum-appdynamics.com'; config.adrumExtUrlHttp = 'http://cdn.appdynamics.com'; config.adrumExtUrlHttps = 'https://cdn.appdynamics.com'; config.xd = {enable : false}; })(window['adrum-config'] || (window['adrum-config'] = {})); window['adrum-config'] = { userEventInfo: { "PageView": function(context) { return{ userData: { page_title: "RandomGeneratedTest" } } } } }; This will make changes to the default JavaScript Agent code that is injected into the page source, allowing the user to see custom data addition in the page source. ... View more

Problem EUM service fails to start with the following error in the EUM server log: 29 Jan 2018 12:02:32.842 +0000 main dropwizard.ServerCommand ERROR | Unable to start server, shutting down | MultiException[java.security.UnrecoverableKeyException: Cannot recover key, java.security.UnrecoverableKeyException: Cannot recover key, java.util.concurrent.RejectedExecutionException: org.eclipse.jetty.util.thread.NonBlockingThread Analysis The password for the key-store specified in the EUM property is incorrect or it does not match the private key password. For EUM SSL implementation, the key-store password and the private key password must be the same.  Solution Ensure that the password for the keystore and the private key match, and are specified in the eum.properties file. To change the private key password using the keytool command: 1. Take a backup of the EUM keystore. 2. Run the following command: <Path to JAVA/bin>/keytool -keypasswd  -alias [Alias name for private key]  -keystore [path to key store] 3. Ensure that the new key password is the same as the key-store password. Verify the same password is being specified in the eum.properties file as well. 4. Restart EUM after making these changes. ... View more