What user permissions must be granted when installing Java Agent binaries? While installing Java Agents, it is mandatory for the user running JVM to have certain access permissions for agent binaries. In this article, find an approach for assigning these permissions, which can be validated in the best order for the environment in which Agents are being installed.  In this article...  How do I assign mandatory permissions for agent binaries? Examples  Additional resources  How do I grant required user permissions when installing agent binaries?  While installing Java agents, it is mandatory for the user running JVM to have certain access permissions to agent binaries.   The following approach can be validated for use in the best order for the environment in which Agents are being installed. For example, you can make slight adjustments based on the OS version, type, and allowed security privileges at the environmental level.  The user must have write privileges to the  conf  and  logs  directories in the Java Agent home. One way to achieve this is to install the agent as the same user that owns the JVM. Provide admin or 777 to agent binaries recursively. Writable permissions to conf/logs and read permissions for all files recursively. Executable permissions to the javaagent.jar file being referenced in the installation. For example: For RHEL9 with SE Linux turned ON Executable permissions to javaagent.jar will be needed for all users. ( chmod a+rx <path>/javaagent.jar ) For IBM WAS running on AIX/UNIX/Linux Try set agent binaries user as JVM/WAS running user to avoid user permissions conflicts.   Additional resources See Install the Java Agent in the documentation  ... View more

Question Is AppDynamics able to monitor MemSQL DB?  Answer Yes. MemSQL is wire-compatible with MySQL. MemSQL uses the MySQL jdbc libraries for connectivity. AppDynamics database agent can monitor the MemSQL DB if configured like this: 1. Install database agent     https://docs.appdynamics.com/display/PRO43/Install+the+Database+Agent 2. Configure a collector using the following details: (refer memsql-collector.png)  https://docs.appdynamics.com/display/PRO43/Configure+MySQL+Collectors          Database Type - MySQL          Host - MemSQL db host          Port - MemSQL db listener port          user name & password - should be entered 3. User permissions should be as mentioned in the document: https://docs.appdynamics.com/display/PRO43/Configure+MySQL+Collectors#ConfigureMySQLCollectors-permisUserPermissionsforMySQL 4. Upon successful installation and configuration, the data will be reported as shown in the following screenshots.   Note the above is tested with following artifacts: 1. MemSQL Enterprise Trial version 2. Database agent 4.3.1.x 3. Controller version 4.3.3.x References: https://en.wikipedia.org/wiki/MemSQL https://www.memsql.com/content/architecture https://docs.memsql.com/ ______________________________________ Was this helpful? Give it kudos [click on the thumbs up icon at the end of the article, bottom left corner] to let others know too! Do you have a question or need clarification? [Post Your Comment] Below.   ... View more

Instrumenting Tableau Server (through version 10.5) for Java Agent   Tableau Server installer spawns a number of processes. You’ll need to identify which processes to monitor based on your need. For example, to monitor these processes execute the following commands to attach the Java Agent to the intended Java processes. clustercontroller searchserver backgrounder vizportal vizqlserver   Make sure that each Java process (node) is associated to a different tier.   Example: - Dappdynamics.agent.tierName=vizportal Step 1 - Edit these 5 JVM options to attach Java Agent tab-install-dir/bin>tabadmin set vizqlserver.vmopts "'-XX:+UseConcMarkSweepGC -XX:NewRatio=2 -XX:SurvivorRatio=6 -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=15 -javaagent:C:\vizqlserver\javaagent.jar -Dappdynamics.agent.tierName=vizqlserver'" tab-install-dir/bin>tabadmin set vizportal.vmopts "'-XX:+UseConcMarkSweepGC -XX:+ExitOnOutOfMemoryError -javaagent:C:\vizportal\javaagent.jar -Dappdynamics.agent.tierName=vizportal'" tab-install-dir/bin>tabadmin set backgrounder.vmopts "'-XX:+UseConcMarkSweepGC -Xmx512m -Xms256m -javaagent:C:\backgrounder\javaagent.jar -Dappdynamics.agent.tierName=backgrounder’” tab-install-dir/bin>tabadmin set clustercontroller.vmopts "'-XX:+UseConcMarkSweepGC -Xmx512m -Xms64m -Djna.nosys=true -XX:+ExitOnOutOfMemoryError -javaagent:C:\Anka\4\ver4.3.0.2\javaagent.jar'" tab-install-dir/bin>tabadmin set searchserver.vmopts "'-XX:+UseConcMarkSweepGC -Xmx512m -Xms256m -XX:+ExitOnOutOfMemoryError -javaagent:C:\Anka\1\ver4.3.0.2\javaagent.jar'" For each set command execution, the result should be shown as below to indicate the successful update of arguments. -- value for key <process-name>.vmopts has been set Example: -- value for key vizportal.vmopts has been set Step 2 - Flush configuration to all the gateways of the Tableau server tab-install-dir/bin>tabadmin set config Expected output without errors: ===== Wrote configuration files to C:/ProgramData/Tableau/Tableau Server/data/tabsvc/config -- Web data connectors were distributed to all gateways. Step 3 - Restart the Tableau server To effect the changes, restart the Tableau server using the appropriate steps, depending on your version of Tableau: How do I restart Tableau Server prior to version 10.5? How do I restart Tableau Server version 10.5 and later?   How do I restart Tableau Server prior to version 10.5? Restart the Tableau server so the changes go into effect: tab-install-dir/bin>tabadmin restart Expected output without any errors: Status: RUNNING On successful instrumentation of the 5 java processes listed in step 1, the Agents should collect and show an application flow map similar to this:  (refer app-dashboard.png).   Potential issues while attaching the agent to the Java process. If the Java Agent does not attach properly to the Java process, you may see a process startup failure, such as: tab-install-dir/bin>tabadmin set vizportal.vmopts “-javaagent:C:\app\4.3.5\ver4.3.5\vizportal\javaagent.jar’” tab-install-dir/bin>tabadmin config Execution shows similarly to: *** Configuration used: -XX:+UseConcMarkSweepGC -XX:+ExitOnOutOfMemoryError -javaagent:C:\app\4.3.5… -Xmx512m -Xms256m   In the above example, the Java Agent location is not properly taken by Tableau when updating the JVM arguments. To fix the error, make sure that the Java Agent location doesn’t contain “-” (hyphen), “.”(dot), or “_” (underscore). The example command should work properly if formatted correctly: Reference Sample Cmd   tab-install-dir/bin>tabadmin set vizportal.vmopts "-javaagent:C:\vizportal\javaagent.jar'" tab-install-dir/bin>tabadmin config   Execution should show something similar to: *** Configuration used: -XX:+UseConcMarkSweepGC -XX:+ExitOnOutOfMemoryError -javaagent:C:\vizportal\javaagent.jar -Xmx512m -Xms256m How do I restart Tableau Server version 10.5+? As of Tableau version 10.5, the tabadmin utility deprecated along with the Tableau Server Configuration utility was replaced by Tableau Services Manager (TSM). If you’re using Tableau 10.5+, use something similar to the following with TSM to restart the server, so the changes go into effect: <Tableau_Home>\bin> tsm configuration set -k vizportal.vmopts -v "-javaagent:C:\vizportal\javaagent.jar'" For more information, see https://help.tableau.com/current/server/en-us/cli_configuration-set_tsm.htm ... View more

An admin user can configure a regular Windows user to access Windows Management Instrumentation (WMI) information by adding the regular user account to the Distributed COM Users and the Performance Monitor Users group (using  lusrmgr.msc ), and then configure the DCOM security settings to allow the groups to access the system remotely (using  dcomcnfg ). Steps for Windows 2003 R2 SP2 Server & Windows 2008 R2 Datacenter: Click Start > Run..., type  lusrmgr.msc  and click OK. In the Users folder, right click the user to bring up the menu, and select Properties. Click on the Member Of tab, and click Add... Under "Enter the object names to select", add the Distributed COM Users group, click Check Names, and then click OK. Click Add... Repeat step 4 for the Performance Monitor Users group. Next, configure the DCOM Security Settings to allow the groups to access the system remotely. Click Start > Run..., type dcomcnfg and click OK. Drill down into the Component Services tree until you get to My Computer. Right-click "My Computer" to bring up the menu, and click Properties. Click the COM Security tab, then click Edit Limits under the Launch and Activation Permissions section. Click Add... Under Enter the object names to select, type Distributed COM Users, click Check Names, then click OK. Click Add... Under Enter the object names to select, type Performance Monitor Users, click Check Names, then click OK. Check Allow for each of the permissions (Local Launch, Remote Launch, Local Activation, Remote Activation) for each of these groups, and click OK. Finally, set the WMI Control security settings to be applied to all namespaces. Click Start > Run..., type wmimgmt.msc and click OK Right-click WMI Control (Local) to bring up the menu, and click Properties. Click over to the Security tab, then click Root, and click the Security button. Click Add... Under Enter the object names to select, type Distributed COM Users, click Check Names, then click OK. Click Advanced. Highlight the row with Distributed COM Users in it and click Edit... From the drop-down list, select "This namespace and sub namespaces" Under the Allow column check Execute Methods, Enable Account, and Remote Enable. Repeat steps 12-17 for the Performance Monitor Users group. Click OK to close all windows. NOTE: If you are using Windows Server 2003 SP1 or later, you will have to run the following steps to access the Win32_Service class due to a known issue (http://support.microsoft.com/kb/907460). Click Start > Run..., type cmd and click OK. Type the following command at the command prompt and then press Enter: sc sdset SCMANAGER D:(A;;CCLCRPRC;;;AU)(A;;CCLCRPWPRC;;;SY)(A;;KA;;;BA)S:(AU;FA;KA;;;WD)(AU;OIIOFA;GA;;;WD) You can now perform WMI monitoring on this Windows host with a regular user account instead of an admin account.   Alternatively, in simple words: Administrative Tools > Component Services > Computers > My Computer > Properties > COM Security Access Permissions > Edit Limits > add user + Local Access + Remote Access Launch and Activation Permissions > Edit Limits > add user + Local Access + Remote Access wmimgmt.msc On the Security tab, highlight Root/CIMV2, click Security > add user and enable the options: Execute Methods, Enable Account and Remote Enable. The above permissions work for Windows 2012 as well. Verify WMI connectivity: Once the user is able to connect using the WMI test, the DB agent can connect and monitor the data. ... View more

Problem: The user is unable to view queries and query wait states information in the Controller UI, even though the information has been captured and sent to the Controller by the Database Agent. Database logs show the following error messages: Example 1: [Event-Uploader-Scheduler-1] 31 May 2017 16:56:09,956 WARN DBMeasurementUploader - Fail to upload Query Stats Measurements: com.singularity.ee.rest.ResponseReadException: Error in controller in processing binary request Write Query Stats - null at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.<init>(ABinaryControllerRequest.java:65) at com.singularity.ee.rest.controller.request.dbmon.WriteQueryStatsRequest.<init (WriteQueryStatsRequest.java:30) at com.singularity.ee.agent.dbagent.task.reporter.DBMeasurementUploader.uploadrQueryStats(DBMeasurementUploader.java:119) Example 2: [Event-Uploader-Scheduler-2] 13 Jul 2017 15:18:09,356 WARN DBMeasurementUploader - Fail to upload Wait Time Measurements: com.singularity.ee.rest.ResponseReadException: Error in controller in processing binary request Wait Time Measurement - null at com.singularity.ee.rest.controller.request.ABinaryControllerRequest.<init>(ABinaryControllerRequest.java:65) at com.singularity.ee.rest.controller.request.dbmon.WaitTimeMeasurementCollectorRequest.<init>(WaitTimeMeasurementCollectorRequest.java:34) The corresponding Controller server.log file may show the following messages: [#|2017-07-13T14:46:01.397+0500|WARNING|glassfish4.1|com.appdynamics.dbmon.core.writer.MeasurementWriter|_ThreadID=368;_ThreadName=http-listener-1(5);_TimeMillis=1499939161397;_LevelValue=900;|Failed to persist 'dbmon-query-stat' measurements for account'customer1_39714636-eec0-4331-932d-1394d05fabd3'. Marking account for reregistration.|#] [#|2017-07-13T14:46:01.397+0500|SEVERE|glassfish4.1|com.singularity.ee.controller.beans.ExceptionHandlingInterceptor|_ThreadID=368;_ThreadName=http-listener-1(5);_TimeMillis=1499939161397;_LevelValue=1000;Encountered runtime exception RestException(statusCode=406, code=Expired.EventType, errorMessage=Eventtype [dbmon-query-stat] for account [customer1_39714636-eec0-4331-932d-1394d05fabd3] expired on [2017-06-30T07:00:00.000Z], developerMessage=Event type[dbmon-query-stat] for account[customer1_39714636-eec0-4331-932d-1394d05fabd3] expired on[2017-06-30T07:00:00.000Z]) at com.appdynamics.analytics.shared.rest.exceptions.RestExceptionFactory.makeException(RestExceptionFactory.java:47) [#|2017-07-13T14:46:01.365+0500|WARNING|glassfish 4.1|com.appdynamics.dbmon.core.writer.MeasurementWriter|_ThreadID=369;_ThreadName=http-listener-1(6);_TimeMillis=1499939161365;_LevelValue=900;|Failed to persist 'dbmon-wait-time' measurements for account 'customer1_39714636-eec0-4331-932d-1394d05fabd3'. Marking account forreregistration.|#] [#|2017-07-13T14:46:01.366+0500|SEVERE|glassfish4.1|com.singularity.ee.controller.beans.ExceptionHandlingInterceptor|_ThreadID=369;_ThreadName=http-listener-1(6);_TimeMillis=1499939161366;_LevelValue=1000;|Encountered runtime exception RestException(statusCode=406, code=Expired.EventType, errorMessage=Eventtype [dbmon-wait-time] for account [customer1_39714636-eec0-4331-932d-1394d05fabd3] expired on[2017-06-30T07:00:00.000Z], developerMessage=Event type[dbmon-wait-time] for account[customer1_39714636-eec0-4331-932d-1394d05fabd3] expired on[2017-06-30T07:00:00.000Z]) at com.appdynamics.analytics.shared.rest .exceptions.RestExceptionFactory.makeException(RestExceptionFactory.java:47)   Solution: On-Premise Controllers: Execute the attached file (expiryfix.sh) by providing the following details: Controller Key: This value is located in the controller admin.jsp page under the Controller Settings tab, associated with the key: appdynamics.on.premise.event.service.key Global Account Name: This value is located in the admin.jsp page under the Accounts tab. Search and select the account to find the Global Account Name value for that account. Expiration Date: To find this value, execute the following SQL query on the Controller database, located under lt;controller-install-dir>/bin$ ./controller.sh login-db . select expiration_date from account where name='<account name>';  Note: Ideally the account name is "customer1" unless using a multi-tenant Controller. The returned value will be a timestamp in milliseconds. Convert this value to Zulu Time Format. (For example, 2017-07-15T07:00:00.000Z)  The URL http://localhost:9080 in the attached script should be replaced by the actual Events Service host name & port number. SaaS Controllers: Contact AppDynamics support. ... View more

  Problem: Scenario 1: The Agent stops collecting or sending metrics to the Controller intermittently, but the Agent is shown as "running." All, or nearly all collectors are temporarily disabled via the UI. Agent logs include error messages similar to the following: [<Athene Database - Prod>-Scheduler-1] 14 May 2017 09:08:28,910  INFO ADBCollector - DB Collector Athene Database - Prod is temporarily disabled. 

[<35111_CDW_DB_PROD>-Monitor-Scheduler-3] 29 Jun 2017 00:59:27,047 ERROR ManagedMonitorDelegate - Fatal transport error while connecting to URL [/controller/instance/317/events]: java.net.SocketException: Broken pipe

 Scenario 2: The Database Agent has crashed abruptly, or has crashed with Out Of Memory
 errors.   Solution: 1. First, determine if the Database Agent is running with heap settings. Database Agent logs should contain a parameter similar to  -Xmx1536m in the  INFO DBAgent - JVM Args: section. If this parameter is not listed, the user needs to add the heap settings to the startup script for the Agent. Example: java -Xmx1536m -Djava.library.path=E:\dbagent_Prod\auth\x64 -Ddbagent.name=FCA -Database -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError=taskkill /F /PID %p -jar db-agent.jar 

2. Once the heap settings are provided, then check if the settings match the Database Agent systems requirements.   Documentation: Database Visibility Requirements and Supported Environments   Recommendation: 1 GB of heap space and an additional 256 MB of heap space for each collector. When monitoring production databases, the heap usage per collector may reach 512 MB.
    
 3. Increase the heap value based on the number of collectors.
   Example: Monitoring 5 databases instances requires ( 5 x 256 MB ) + 1024 MB = 2,304 MB 
4. Database Agent restart is required for the above changes. ... View more

When the Java or Database Agent stops communicating, or cannot communicate, with the Controller, and an error message appears in the agent logs, how do I disable the SSL certification validation between the agent and Controller? In this article... Symptom | Troubleshooting and solution  | More about this JVM flag Symptom You observe the following error message is visible in the Agent logs: [Thread-0] 22 Jun 2017 11:53:52,467 ERROR ConfigurationChannel - Exception: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Troubleshooting and solution Verify whether the SSL certificate is installed and enabled by following the steps in the documentation: Enable SSL for the Java Agent Alternatively, run the Java Agent with the following system property to resolve the issue: -Dappdynamics.force.default.ssl.certificate.validation=false More information about this JVM flag: appdynamics.force.default.ssl.certificate.validation=false This JVM flag means that minimal certificate validation is done, which means that the notBefore   and notAfter are still checked. If this property is set to true, full certification chain validation is done.  There is no way to fully disable the validation of notBefore and notAfter dates as that defies the purpose of SSL.  When this property is set to false, the  X509Certificate.checkValidity method is called, which would validate the certificate start and expiry dates.  https://docs.oracle.com/javase/8/docs/api/java/security/cert/X509Certificate.html#checkValidity-- ... View more

Users utilizing version 4.3.x Database Agents may experience an issue where the database Collector does not start, and no longer reports metrics to the Controller. The Database Agent logs will show one of the following errors for the problematic Collector: Case-1: java.sql.SQLException: ORA-24247: network access denied by access control list (ACL) Case-2: java.sql.SQLSyntaxErrorException: ORA-00904: : invalid identifier, event after applying the Case-1 solution   To resolve case 1, follow the permissions instructions within the documentation for configuring the Oracle RAC. Another option would be to add the following JVM property to the custom agent build (for custom build information, contact support) if uninterested in making changes to the database permissions. -Ddbagent.oracle.rac.use.hostname=true For case 2, run the following SQL command in the Oracle cluster environment. select HOST_NAME from gv$instance; Then validate network connectivity. The Database Agent's machine must be able to TelNet on the listener port (configured for the Collector) for all hosts that are returned from the above query.  Notes: As of version 4.3.x, only one Collector should be sufficient to monitor an entire Oracle cluster. The Database Agent and Collector user should have access to all the nodes in the Oracle cluster. If the Database Agent has been upgraded from 4.x to 4.3 then only a single Collector should be enough, the remaining Collectors can be deleted. Once a Collector is deleted then all historical data will be lost. It is not possible to restore the deleted Collectors. ... View more

What do I need to know to set up a Database Agent with High Availability? In this article  Starting principles when using High Availability (HA) Agent location and naming for HA Crucial considerations for configuration Additional Resources Starting principles The purpose of running agents in High Availability (HA) is to minimize disruption. When one agent fails or is experiencing downtime, then the other agent can continue to monitor and report. At any given time, one Agent will be active while the others are in passive states. If one Agent goes down, the other Agents will, in turn, pick up the Database Collectors and begin reporting. Agent location and naming for HA To set up a Database Agent with High Availability (HA), we recommend installing each Agent on a different host. If the Agents must be on the same host, then they need to be installed into different folders.  The primary and secondary Database Agent should both have exactly the same name. For example,  -Ddbagent.name=<same-name> . DB AGENT LOCATION HOST Primary (host 1) $ java -Ddbagent.name=abc -jar db-agent.jar Secondary (host 2) $ java -Ddbagent.name=abc -jar db-agent.jar Start the agents using the steps in Start and Stop the Documentation. Crucial considerations for configuration  Keep in mind that all Database Agents in the HA configuration should: Be running at all times — If the Agents are not running, then the Agents will not be visible on the Controller Have access to the monitoring databases which are defined in the Collector configuration In addition, be sure that Collectors are configured to use the active Agent. Additional Resources Controller High Availability Using the High Availability Toolkit Database Agent Configuration Properties ... View more

For users whose MySQL database collector will not start, first check the configuration data including username, password, host, and port. Look in the Agent logs for an error similar to the following: "Error collecting data for database 'xxx DB' java.sql.SQLException: Access denied for user 'root'@'hostname'" at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1075) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3566) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3498) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:919) at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:4004) at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1284) To diagnose this issue, first validate the following: Telnet to the MySQL database from the Database Agent host with the data provided in the collector. ($telnet hostname port) If the Telnet works but the collector returns an "Access denied" error message, follow the next steps. Run the following command in MySQL database: mysql> select user, host from mysql.user where name = 'root'; Note: root is the user for which the error occurred in the collector start up. Validate whether the user that is mapped to the hostname provided in the collector configuration matches the output(host) of the previous command. Both the hostnames should match. Resolve the issue by either updating the collector configuration to match the MySQL command output from above, or create a new hostname that matches the collector configuration output and use this new hostname in the collector configuration.    Related Links: Configure MySQL Collectors MySql: Give Root User Logon Permission From Any Host   ... View more