All TKB Articles in Learn Splunk

Learn Splunk

All TKB Articles in Learn Splunk

The Events Service cluster may not function correctly and its health state will be listed as red if there are unwanted indices either in the unassigned or initializing state. Example: In a br... See more...
The Events Service cluster may not function correctly and its health state will be listed as red if there are unwanted indices either in the unassigned or initializing state. Example: In a browser or terminal, run the following: curl http://localhost:9081/healthcheck?pretty=true The output of the healthcheck command will show cluster state [RED] . "events-service-api-store / elasticsearch-singlenode-module" : { "healthy" : true, "message" : "Current [appdynamics-events-service-cluster] cluster state: [RED], data nodes: [1], nodes: [1], active shards: [48], relocating shards: [0], initializing shards: [0], unassigned shards: [5], timed out: [false]" }, These unwanted indices are created as a result of the Nessus Vulnerability Scanner that gets run on the cluster port (default 9200). Diagnosis: Run the following command on the Events Service machine or in a browser to verify if there are unwanted indices. http://<events_service_machine>:9200/_cat/shards?v [in Browser] OR curl http://<events_service_machine>:9200/_cat/shards?v [in the terminal of the Events Service machine] Output example: index shard prirep state docs store ip node vrvsgs1.html 4 p STARTED 0 114b 192.x.y.z node-xyzxyzxyz vrvsgs1.html 4 r UNASSIGNED 0 168b 192.x.y.z node-xyzxyzxyz vrvsgs1.html 0 r UNASSIGNED 0 791b 192.x.y.z node-xyzxyzxyz vrvsgs1.html 0 p STARTED 0 125b 192.x.y.z node-xyzxyzxyz vrvsgs1.html 3 r STARTED 0 105b 192.x.y.z node-xyzxyzxyz gvanws1.html 4 p STARTED 0 125b 192.x.y.z node-xyzxyzxyz gvanws1.html 4 r STARTED 0 128b 192.x.y.z node-xyzxyzxyz gvanws1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz gvanws1.html 0 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz gvanws1.html 3 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz oapssw1.html 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz oapssw1.html 0 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz oapssw1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz nrawos1.html 2 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz nrawos1.html 2 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz nrawos1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz nrawos1.html 0 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz spipe 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz spipe 4 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz spipe 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz spipe 0 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz psaoon1.html 2 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz psaoon1.html 2 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz psaoon1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz psaoon1.html 0 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz arenon1.html 4 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz arenon1.html 0 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz arenon1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz arenon1.html 3 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz sprreo1.html 1 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz sprreo1.html 2 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz sprreo1.html 2 p UNASSIGNED 0 115b 192.x.y.z node-xyzxyzxyz vepper1.html 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz vepper1.html 4 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz vepper1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz perl 1 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz perl 2 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz perl 2 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz gaowsa1.html 4 p UNASSIGNED 0 115b 192.x.y.z node-xyzxyzxyz greprs1.html 4 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz greprs1.html 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz enoson1.html 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz enoson1.html 4 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz enoson1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz vpnwwg1.html 2 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz vpnwwg1.html 2 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz arenon1.html 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz arenon1.html 4 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz soaeap1.html 3 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz soaeap1.html 1 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz pwvwgg1.html 1 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz pwvwgg1.html 2 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz pwvwgg1.html 2 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz enawae1.html 4 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz enawae1.html 4 p UNASSIGNED 0 115b 192.x.y.z node-xyzxyzxyz sawmill6cl.exe 4 p UNASSIGNED 0 115b 192.x.y.z node-xyzxyzxyz sawmill6cl.exe 4 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz rogwsw1.html 4 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz Solution:  Delete these unwanted indices using the following commands: Delete all unwanted indices using the following curl command in the machine terminal: curl -XDELETE http://<events-service_hostname>:9200/<index_name> Example: curl -XDELETE http://localhost:9200/rogwsw1.html Verify if unwanted indices have been removed using the following commands: curl http://<events_service_machine>:9200/_cat/shards?v curl http://<events_service_machine>:9200/_cat/indices?v Work with a network or security team to configure the Nessus Vulnerability Scanner toexclude scanning the Events Service cluster port (9200-9300).
Instrument the Events Service process for performance tuning using the following steps:  Configure the Java Agent via the  controller-info.xml  file located in  /AppD/AppServerAgent-4.x.x/ver4.x... See more...
Instrument the Events Service process for performance tuning using the following steps:  Configure the Java Agent via the  controller-info.xml  file located in  /AppD/AppServerAgent-4.x.x/ver4.x.x.0/conf . Add the following option to the  <Events Service home directory>/conf/events-service.vmoptions  file. -javaagent:/AppD/AppServerAgent-4.x.x/ver4.x.x.0/javaagent.jar in conf/events-service.vmoptions Restart the Events Service process. Verify from the Controller UI that you are able to see the Events Service business transactions have registered successfully. Example:
Use the Analytics Agent health check URL in order to check the flow of business transactions and logs, as well as the connection status from the server where the Analytic Agent is hosted to the Eve... See more...
Use the Analytics Agent health check URL in order to check the flow of business transactions and logs, as well as the connection status from the server where the Analytic Agent is hosted to the Events Service. Using curl on the command line, enter the health check URL and append the string  ?pretty=true  to format the output for readability. Example: curl http://localhost:9091/healthcheck?pretty=true Note: Replace "localhost" with your public IP/DNS if you are accessing it from a different machine where the Analytic Agent is installed. The default Analytics Agent admin port is 9091. Sample output: curl http://localhost:9091/healthcheck?pretty=true { "analytics-agent / Build information" : { "healthy" : true, "message" : "buildName=108-4.3.0.next-build, buildHash=8992cf933916e227b6e7ac82a2f25bb15e6ef8d1, buildTimestamp=2017-03-09T01:20:14+0000, buildNumber=7902, jobName=build-analytics-shared, version=4.3.0.0" }, "analytics-agent / Configuration properties" : { "healthy" : true, "message" : "Dynamic properties: []" }, "analytics-agent / Connection to [https://analytics.api.appdynamics.com:443]" : { "healthy" : true }, "analytics-agent / Log Sources" : { "healthy" : true, "message" : "" }, "analytics-agent / http:eventType:publish:bt" : { "healthy" : true, "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]" }, "analytics-agent / http:eventType:publish:log" : { "healthy" : true, "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]" }, "analytics-agent / http:receive:bt[[/v1/sinks/bt/*, /v2/sinks/bt/*]]" : { "healthy" : true, "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]" }, "analytics-agent / pipelines" : { "healthy" : true, "message" : "[16] pipeline(s). bt-publish-0: [MessageSourceStage: 1, EventServicePublishStage: 0], bt-publish-1: [MessageSourceStage: 1, EventServicePublishStage: 0], bt-publish-2: [MessageSourceStage: 1, EventServicePublishStage: 0], bt-publish-3: [MessageSourceStage: 1, EventServicePublishStage: 0], bt-publish-4: [MessageSourceStage: 1, EventServicePublishStage: 0], bt-publish-5: [MessageSourceStage: 1, EventServicePublishStage: 0], bt-publish-6: [MessageSourceStage: 1, EventServicePublishStage: 0], bt-publish-7: [MessageSourceStage: 1, EventServicePublishStage: 0], log-publish-7: [MessageSourceStage: 1, EventServicePublishStage: 0], log-publish-6: [MessageSourceStage: 1, EventServicePublishStage: 0], log-publish-5: [MessageSourceStage: 1, EventServicePublishStage: 0], log-publish-4: [MessageSourceStage: 1, EventServicePublishStage: 0], log-publish-3: [MessageSourceStage: 1, EventServicePublishStage: 0], log-publish-2: [MessageSourceStage: 1, EventServicePublishStage: 0], log-publish-1: [MessageSourceStage: 1, EventServicePublishStage: 0], log-publish-0: [MessageSourceStage: 1, EventServicePublishStage: 0]" }, "analytics-agent / queues" : { "healthy" : true, "message" : "[2] queues [[biz_txn_v1] ratio: [0.00], size: [0], capacity: [128], [log_v1] ratio: [0.00], size: [0], capacity: [128]]" }, "deadlocks" : { "healthy" : true } Elements  Details analytics-agent / Build information Top level, if healthy indicates all sub-components are healthy as well. Contains the version number as well. "analytics-agent / Connection to [https://analytics.api.appdynamics.com:443]" Where is the Analytics Connecting to and in this case its SAAS events-service API analytics-agent / http:eventType:publish:bt This is a method which processes the incoming business transaction analytics data and sends it to the Events Service. 0.0000000 means no activity. 1.000000 means maxed out. Average over last five minutes. analytics-agent / http:receive:bt[/v1/sinks/bt/*] This is a method to which data is received from server agents configured to send business transaction analytics data. Average over last five minutes. analytics-agent / pipelines 16 pipelines total (current default). These processes the messages to be sent to the publish service. analytics-agent / queues Queue size default is 128. biz_txn_v1 refers to the queue for business transaction analytics data. If the queue is full, could be indicative of network and auth issues.     Note: If there's an authentication issue while connecting to Events Service,  the output would show as below: }, "analytics-agent / Connection to [https://analytics.api.appdynamics.com:443]" : { "healthy" : false, "message" : "Status code: [401], Message: The supplied auth information is incorrect or not authorized." To resolve the authentication issue, check that the event name, account name, and access key within the Analytic Agent properties file are set up correctly. Refer to the following link for details: Installing Agent Side Components. Related Links: Troubleshooting Analytics Agent Issues
Use the health check URL on the Events Service admin port in order to get the status (green, yellow, or red) of the Events Service cluster. In the command line using curl, enter the <localhost:90... See more...
Use the health check URL on the Events Service admin port in order to get the status (green, yellow, or red) of the Events Service cluster. In the command line using curl, enter the <localhost:9081>/healthcheck URL and append the string ?pretty=true to format the output for readability. Example: curl http://localhost:9081/healthcheck?pretty=true Note: Replace "localhost" with your public IP/DNS. The default admin port is 9081. Output example:    curl http://localhost:9081/healthcheck?pretty=true { "deadlocks" : { "healthy" : true }, "events-service-api-store / BizOutcome [stage.extract]" : { "healthy" : true, "message" : "Rates (Avg per second. Avg of last 5 min) accepted events: [0.000000], discarded events: [0.000000], acceptance ratio: [0.000000]" }, "events-service-api-store / BizOutcome [stage.filter]" : { "healthy" : true, "message" : "Rates (Avg per second. Avg of last 5 min) accepted events: [0.000000], discarded events: [0.000000], acceptance ratio: [0.000000]" }, "events-service-api-store / Build information" : { "healthy" : true, "message" : "buildName=123-4.3.0.next-build, buildHash=d0e4fc0167eff6b5fa974d1f9a8180e5a1e89044, buildTimestamp=2017-03-17T22:06:29+0000, buildNumber=3709, jobName=build-events-service, version=4.3.0.1" }, "events-service-api-store / Chunking.AllChunksTimer [CatchAllChunkingStrategy]" : { "healthy" : true, "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]" }, "events-service-api-store / Chunking.AllChunksTimer [SeriesChunkingStrategy]" : { "healthy" : true, "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]" }, "events-service-api-store / Chunking.AllChunksTimer [TimeRangeChunkingStrategy]" : { "healthy" : true, "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]" }, "events-service-api-store / Chunking.Counts [CatchAllChunkingStrategy]" : { "healthy" : true, "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]" }, "events-service-api-store / Chunking.Counts [SeriesChunkingStrategy]" : { "healthy" : true, "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]" }, "events-service-api-store / Chunking.Counts [TimeRangeChunkingStrategy]" : { "healthy" : true, "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]" }, "events-service-api-store / Chunking.FirstChunkTimer [CatchAllChunkingStrategy]" : { "healthy" : true, "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]" }, "events-service-api-store / Chunking.FirstChunkTimer [SeriesChunkingStrategy]" : { "healthy" : true, "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]" }, ............................... "events-service-api-store / Resource [v3/events/{eventType}/usage - documents]" : { "healthy" : true, "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]" }, "events-service-api-store / Rolling index management" : { "healthy" : true, "message" : "Healthcheck for [Rolling index management]: Is running flag is: [false], last execution successful flag is: [true]" }, "events-service-api-store / elasticsearch-singlenode-module" : { "healthy" : true, "message" : "Current [appdynamics-events-service-cluster] cluster state: [GREEN], data nodes: [1], nodes: [1], active shards: [11], relocating shards: [0], initializing shards: [0], unassigned shards: [0], timed out: [false]" }, "events-service-api-store / pipelines" : { "healthy" : true, "message" : "[1] pipeline(s). biz-outcome: [MessageSourceStage: 1, FilterStage: 0, ExtractStage: 0, UpsertStage: 0]" }, "events-service-api-store / queues" : { "healthy" : true, "message" : "[1] queues [[biz-outcome-incoming-events] ratio: [0.00], size: [0], capacity: [1000]]" } The above example shows the cluster state is green, which means fully functional with no unassigned, relocating or initialising shards. "events-service-api-store / elasticsearch-singlenode-module" : { "healthy" : true, "message" : "Current [appdynamics-events-service-cluster] cluster state: [GREEN], data nodes: [1], nodes: [1], active shards: [11], relocating shards: [0], initializing shards: [0], unassigned shards: [0], timed out: [false]" }, To check the connectivity, use the ping URL on the Events Service default port (9080). A "pong" message means success.  curl http://localhost:9080/_ping _pong
In a large MySQL database with numerous tables and partitions, the  information_schema.partitions table is built serially and is cached once built. This is so the App Server can be restarted quick... See more...
In a large MySQL database with numerous tables and partitions, the  information_schema.partitions table is built serially and is cached once built. This is so the App Server can be restarted quickly if the database is not also restarted. For this reason, it is not recommended to restart the database unless absolutely necessary. To track how long the Controller takes to start, run the following command from the computer terminal during startup repeatedly at two-minute intervals until the Controller UI is accessible: echo "select time, info from information_schema.processlist where command = 'Query' and user != 'root';" | <controller_home>/bin/controller.sh login-db  If the results include the  information_schema.partitions , the Controller is building the list of partitions and the Controller UI would not be accessible yet. If frequent database restarts are necessary, the amount of time to start can be improved by adding the innodb_stats_sample_pages=1 property to the end of the <controller_home>/db/db.cnf file (after "# End of standard db.cnf" ) if it does not exist already and restarting the Controller. This property is added by default for High Availability Controller Deployments. Example: # End of standard db.cnf # added to speed up startup innodb_stats_sample_pages=1
When the Controller is set up for High Availability, users see a "Controller disabled on this host" message on the secondary Controller when a partial replication is run and full replication with t... See more...
When the Controller is set up for High Availability, users see a "Controller disabled on this host" message on the secondary Controller when a partial replication is run and full replication with the -f flag is not yet complete. When a replication runs without the -f flag, it copies the MySQL data and index files from the primary server to the secondary while App Server (Glassfish) runs on the primary server. There may be tables updated by App Server thus changing the underlying data files, but these files may have already been copied over to the secondary server. This is an incremental sync. Running a replication with the -f flag stops the Controller on the primary server, ensuring that no files will change, then compares the files between the primary and secondary servers and only copies those that are different. This is final sync. More incremental syncs create less downtime when running a replication with the -f flag, as there are fewer files to copy in the direct sync. To ensure MySQL does not start on secondary server until the final sync is complete, replicate script deletes the  <controller_home>/bin/controller.sh  file on the secondary server until the replication is finished and hence the message "Controller disabled on this host" so that controller cannot be started on secondary. Once the final replication completes,<controller_home>/bin/controller.sh is automatically copied from primary to secondary server and MySQL on secondary server is started.
Question Why are there unknown values for Mobile Carriers and Connection Types in Mobile RUM dashboard? Answer The AppDynamics Android Agent references ConnectivityManager and Telep... See more...
Question Why are there unknown values for Mobile Carriers and Connection Types in Mobile RUM dashboard? Answer The AppDynamics Android Agent references ConnectivityManager and TelephonyManager for information on the device's connection and carrier, respectively. Sometimes these frameworks fail to return information, for a variety of reasons, which are beyond the control of AppDynamics. Example: Some devices from specific vendors disallow access to the Connectivity Manager for security reasons, and appear in device logs as "Access to ConnectivityManager is denied". There are Android devices and versions where the Connectivity Manager is not available or does not work in certain hardware configurations. There are several documented cases of this, the specific Android error is "missing IConnectivityManager", and appears in device logs as "Error determining connection type." Frequently, devices without SIM cards, like tablets, will fail to return carrier info from Telephony Manager. In the dev logs, they appear as "Error determining carrier name". Note: A value of "offline" in the dashboard means that connectivity was limited at the time the request was sent Related Links: More on Mobile RUM Mobile RUM metrics Video - Start monitoring with the Android Agent
Users installing their Controller on a Windows machine may see the following error message on the installer screen: Please resolve the errors and retry the operation. A Microsoft Windows hotfi... See more...
Users installing their Controller on a Windows machine may see the following error message on the installer screen: Please resolve the errors and retry the operation. A Microsoft Windows hotfix is required but is not detected on this system. Please install the hotfix from http://support.microsoft.com/kb/2549760 and retry the operation. This hotfix ensures that the windows registry modifications made by this installer to extend the default service timeouts work as expected. If you are unable to install the hotfix, send the file C:\Users<Username?\AppData\Local\Temp\hotfix_status_XXXXXXX.txt and the installation log in your temp directory to AppDynamics support for assistance. The error message appears when hotfix #2549760 has not been installed on the Windows server.  Solution Bypass the hotfix validation during Controller installation: Launch a command prompt as an Administrator on your Windows machine. Run the Controller installer with a -Vskip_validation=true flag to bypass the hotfix check. Example: <controller_installer_name>.exe -Vskip_validation=true Note: There is no loss of functionality without the hotfix, but be aware that until the hotfix is installed, users must stop all Controller services (including app and database) and ensure they have completely shut down before rebooting the Controller. This prevents database corruption which may be irreversible. Installing the hotfix after successful Controller installation: Shutdown all Controller related services (app and database). Install the hotfix by running the following commands in a command prompt as an Administrator: reg add HKLM\SYSTEM\CurrentControlSet\Control /v WaitToKillServiceTimeout /t REG_SZ /d 600000 /f   Increase service stop timeouts reg add HKLM\SYSTEM\CurrentControlSet\Control /v ServicesPipeTimeout /t REG_SZ /d 1200000 /f Increases service startup timeouts Start the controller services.
How do I resolve "Network Communication Error" in the Controller UI? Table of Contents Syptoms Diagnosis Solution Symptoms The following "Network Communication Error" appears in the Cont... See more...
How do I resolve "Network Communication Error" in the Controller UI? Table of Contents Syptoms Diagnosis Solution Symptoms The following "Network Communication Error" appears in the Controller UI: Diagnosis Error message in <controller_home>/logs/server.log : [INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.l ogging|_ThreadID=58;_ThreadName=Thread-5;|[BlazeDS] 05/24/2017 16:07:50.527 [ERROR] [Endpoint.General] Unexpected error encountered in Message Broker servlet flex.messaging.security.SecurityException: Secure endpoint '/messagebroker/amfsecure' must be contacted via a secure protocol. at flex.messaging.endpoints.AbstractEndpoint.validateRequestProtocol(AbstractEndpoint.java:1368) at flex.messaging.endpoints.AbstractEndpoint.service(AbstractEndpoint.java:935) at flex.messaging.endpoints.BaseHTTPEndpoint.service(BaseHTTPEndpoint.java:312) at flex.messaging.MessageBrokerServlet.service(MessageBrokerServlet.java:373) at javax.servlet.http.HttpServlet.service(HttpServlet.java:770) at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1550) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:343 ) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:217) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:89) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130) at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter[INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.l ogging|_ThreadID=58;_ThreadName=Thread-5;|[BlazeDS] [ERROR] [Endpoint.General] Unexpected error encountered in Message Broker servlet flex.messaging.security.SecurityException: Secure endpoint '/messagebroker/amfsecure' must be contacted via a secure protocol. at flex.messaging.endpoints.AbstractEndpoint.validateRequestProtocol(AbstractEndpoint.java:1368) at flex.messaging.endpoints.AbstractEndpoint.service(AbstractEndpoint.java:935) at flex.messaging.endpoints.BaseHTTPEndpoint.service(BaseHTTPEndpoint.java:312) at flex.messaging.MessageBrokerServlet.service(MessageBrokerServlet.java:373) at javax.servlet.http.HttpServlet.service(HttpServlet.java:770) at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1550) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:343 ) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:217) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:89) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130) at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256 ) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:217) at com.singularity.ee.controller.servlet.CsrfFilter.doFilter(CsrfFilter.java:84) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256 ) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:217) at com.singularity.ee.controller.servlet.RequestOriginMarkingFilter.doFilter(RequestOriginMarkingFi lter.java:26) Solution This issue occurs when an SSL-terminating proxy is redirecting requests to the Controller and the Controller is not configured to handle these requests. To resolve this issue, follow the steps int the documentation under "Configure the Controller for SSL Termination at the Proxy."
In the <EUM_HOME>/jre/lib/security/java.security file, add TLSv1 and TLSv1.1 to the existing value of the property  jdk.tls.disabledAlgorithms . Then restart the EUM Server. Example: jdk.tls... See more...
In the <EUM_HOME>/jre/lib/security/java.security file, add TLSv1 and TLSv1.1 to the existing value of the property  jdk.tls.disabledAlgorithms . Then restart the EUM Server. Example: jdk.tls.disabledAlgorithms= SSLv2Hello, SSLv3, TLSv1, TLSv1.1 Note: There must be a space between the comma and next algorithm name. Test the connectivity of the EUM Server using TLSv1 (tls1.0) with OpenSSL. The connectivity should fail with a "handshake failure" error. Example: openssl s_client -connect <eum_host>:<eum_port> -tls1
The time zone IDs available to the Controller Audit History API are Java standard and can be found using getAvailableIDs(). Java example script to find available time zone IDs: import java.ut... See more...
The time zone IDs available to the Controller Audit History API are Java standard and can be found using getAvailableIDs(). Java example script to find available time zone IDs: import java.util.TimeZone; public class TimeZoneIDAvailable { public static void main(String args[]) { String [] Ids = TimeZone.getAvailableIDs(); for (String s: Ids) { System.out.println(s); } } } Note: The time range should only be one day for this API, so start and end times can only be 24 hours apart. Example: The time zone ID for Japan is GMT+0900. The + symbol has to be encoded in curl and is represented as "%2B". curl --user admin@customer1:password http://saradhi:8090/controller/ControllerAuditHistory\?startTime=2017-02-07T06:00:00.607%2B0900\&endTime=2017-02-07T23:50:03.607%2B0900\&timeZoneId=Japan Output: [{"timeStamp":1486449446819,"auditDateTime":"2017-02-07T15:37:26.819+0900","accountName":"customer1","securityProviderType":"INTERNAL","userName":"admin","action":"LOGIN"},{"timeStamp":1486449507278,"auditDateTime":"2017-02-07T15:38:27.278+0900","accountName":"customer1","securityProviderType":"INTERNAL","userName":"admin","action":"LOGIN"},{"timeStamp":1486449551811,"auditDateTime":"2017-02-07T15:39:11.811+0900","accountName":"customer1","securityProviderType":"INTERNAL","userName":"admin","action":"LOGIN"},{"timeStamp":1486450129316,"auditDateTime":"2017-02-07T15:48:49.316+0900","accountName":"customer1","securityProviderType":"INTERNAL","userName":"admin","action":"LOGIN"},{"timeStamp":1486450153704,"auditDateTime":"2017-02-07T15:49:13.704+0900","accountName":"customer1","securityProviderType":"INTERNAL","userName":"admin","action":"LOGIN"},{"timeStamp":1486450181375,"auditDateTime":"2017-02-07T15:49:41.375+0900","accountName":"customer1","securityProviderType":"INTERNAL","userName":"admin","action":"OBJECT_CREATED","objectType":"USER","objectName":"test"},{"timeStamp":1486450184451,"auditDateTime":"2017-02-07T15:49:44.451+0900","accountName":"customer1","securityProviderType":"INTERNAL","userName":"admin","action":"OBJECT_UPDATED","objectType":"USER","objectName":"test"},{"timeStamp":1486450184457,"auditDateTime":"2017-02-07T15:49:44.457+0900","accountName":"customer1","securityProviderType":"INTERNAL","userName":"admin","action":"USER_ADD_ACCOUNT_ROLE","objectType":"USER","objectName":"test"},{"timeStamp":1486450184463,"auditDateTime":"2017-02-07T15:49:44.463+0900","accountName":"customer1","securityProviderType":"INTERNAL","userName":"admin","action":"USER_ADD_ACCOUNT_ROLE","objectType":"ACCOUNT_ROLE"},{"timeStamp":1486450221707,"auditDateTime":"2017-02-07T15:50:21.707+0900","accountName":"customer1","securityProviderType":"INTERNAL","userName":"admin","action":"LOGIN"},{"timeStamp":1486450241029,"auditDateTime":"2017-02-07T15:50:41.029+0900","accountName":"customer1","securityProviderType":"INTERNAL","userName":"admin","action":"LOGIN"}] In the above output, "+0900" is formatted in the timestamp.
Symptoms The AppDynamics Domain1 Service log (located in  <Controller_Home>\appserver\glassfish\domains\domain1\bin\AppDynamicsDomain1Service.err.log ) is growing rapidly when a Controller is ins... See more...
Symptoms The AppDynamics Domain1 Service log (located in  <Controller_Home>\appserver\glassfish\domains\domain1\bin\AppDynamicsDomain1Service.err.log ) is growing rapidly when a Controller is installed on a Windows Server. Diagnosis There is a bug within Glassfish that causes this log file to grow rapidly. This particular file contains the same information as the log files in <controller_home>/logs/server*.log files, so it is possible to rotate the Domain1 Service log without losing any data.  Newer versions of Glassfish include an option to avoid writing any data to the Domain1 Service logs. Solution   Stop the Controller App Server Backup the  Controller\appserver\glassfish\domains\domain1\bin\AppDynamicsDomain1Service.xml file Edit the XML file with the following changes: Change from  <logmode>reset</logmode> to  <logmode>roll</logmode>. Start the App Server. Note:  The log files are rotated with name .old  not dynamically, but only on App Server start and stop. The logs will continue to grow until the next restart. After these steps, the first time the App Server is restarted, the current log file will be renamed to .old . The next time the App Server is restarted, the .old file is purged and the current log file is renamed to .old .  
  If the path (including name) of the agent truststore is specified using the <controller-keystore-filename> element in the agent configuration file  controller-info.xml then agent loads that t... See more...
  If the path (including name) of the agent truststore is specified using the <controller-keystore-filename> element in the agent configuration file  controller-info.xml then agent loads that trustore. If not specified in thecontroller-info.xml, the Java Agent then looks for the truststore file with name cacerts.jks in  <agent_home>/verX.X.X.X/conf directory.  Note: The permissions on the cacerts.jks file needs to be 644. If the cacerts.jks file does not exist in <agent_home>/versionXXXX/conf, but the application specifies truststore using a JVM argument -Djavax.net.ssl.trustStore , then the Java Agent loads that truststore. If the Java Agent can not find the certs in any of the above, it will load the default Java truststore of the JVM being instrumented, <JRE_HOME>/lib/security/ cacerts (password changeit).   Related Links: Java Agent Secure the Java Agent truststore
The property that controls the threshold for the Resource Pool Limit Reached event is  jmx-metric-threshold-percentage , which can be used to change the limit from the default of 80% to a higher or l... See more...
The property that controls the threshold for the Resource Pool Limit Reached event is  jmx-metric-threshold-percentage , which can be used to change the limit from the default of 80% to a higher or lower number. It requires a restart of the agent JVM to apply this change. Example: A user wants to increase the threshold to 90%. In <agent_home>/versionXXXX/conf/app-agent-config.xml add the following property:  <property name="jmx-metric-threshold-percentage" value="90"/> Restart the JVM. Note: On success, in the agent*.0.log the user will see the following message:  DEBUG AgentConfigWatchdog - added service properties [{jmx-metric-threshold-percentage=90....
AppDynamics recommends using fio, a free tool for measuring disk I/O performance. Fio is meant to be used both for benchmark and stress/hardware verification. Fedora maintains builds of fio in thei... See more...
AppDynamics recommends using fio, a free tool for measuring disk I/O performance. Fio is meant to be used both for benchmark and stress/hardware verification. Fedora maintains builds of fio in their Extra Packages on Enterprise Linux (EPEL) repositories. Related Links: AppDynamics Controller System Requirements How to Enable EPEL Repository for RHEL/CentOS 7.x/6.x/5.x Benchmarking using fio How to add EPEL repository to a server: Note: Run the following as root user from the command line. RHEL6 example: Install EPEL repository. wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm rpm -ivh epel-release-6-8.noarch.rpm Install fio using yum yum install fio   RHEL7 example:  Install EPEL repository. wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm rpm -ivh epel-release-7-9.noarch.rpm Install fio using yum yum install fio  EPEL Reference https://fedoraproject.org/wiki/EPEL Published on ‎06-02-2017.
Symptoms On a Linux machine, using High Availability mode on the primary Controller or the single Controller on which service scripts are installed in non-High Availability mode, the command servi... See more...
Symptoms On a Linux machine, using High Availability mode on the primary Controller or the single Controller on which service scripts are installed in non-High Availability mode, the command service appdcontroller start does not start the App Server.  Error message: Skipping appserver start - HA passive Watchdog disabled Diagnosis The MySQL database root user password is not obfuscated and stored. Solution In the computer terminal use the following commands to prompt for the MySQL root password. cd <controller_home>/HA ./save_mysql_passwd.sh Enter the root password, then try the  service appdcontroller start command again. If unsure of the root password, enter the command  <controller_home>/bin/controller.sh login- db and then enter the root password. If the password allows you to log in, it is correct. Stop services: service appdcontroller stop service appdcontroller-db stop Start services: service appdcontroller start See status: service appdcontroller status Related Links: Github - AppDynamics High Availability Toolkit
What are the steps for setting up an alert to be notified when an AppDynamics agent is down? There are three steps to creating an agent-down alert. You will need to set a health rule, create an act... See more...
What are the steps for setting up an alert to be notified when an AppDynamics agent is down? There are three steps to creating an agent-down alert. You will need to set a health rule, create an action, and then create a policy. Contents Set a health rule Create an action to send an email when the Agent is down Create a policy and select the health rule on which the policy should be evaluated   STEP 1  Set a health rule To receive an email alert when an app agent or machine agent is down, you will need to set up a health rule from the Controller UI.  Open Create Health Rule Set the health rule to affect application performance. Input x in "Trigger only when violation occurs x times in the last 10 min(s)" Note: For machine agent availability, select the availability metric under Agent > machine. STEP 2  Create an action to send an email Next, create the action to send an email when the agent is down.  From the Controller UI, create a new action. Enter the email address that should receive the alert. STEP 3  Create a policy and select the health rule Finally, create a new policy and select the health rule you created in STEP 1. From the Controller IU, create a new policy. Select the email address. Additional resources: For instructions on how to set up alerts to monitor the health status of your application, please visit these resources: Health Rules Configuring Health Rules
Finding the load average calculated by a Java Agent on a Windows server requires the following data: ProcessorQueueLength counts how many threads are ready in the processor queue, but not curr... See more...
Finding the load average calculated by a Java Agent on a Windows server requires the following data: ProcessorQueueLength counts how many threads are ready in the processor queue, but not currently able to use the processor. To find this value, launch PowerShell and run the following command: Get-WmiObject -Query "Select * from Win32_PerfRawData_PerfOS_System" | findstr ProcessorQueueLength​ The value of mathematical constant e= 2.718281828459045. Load average intervals are: 1 minute  (60 sec) 5 minutes (300 sec) 15 minutes (900 sec) "Sample frequency" is the frequency with which the samples are collected for each interval. Example: In 1 minute, 2 samples were collected. In 5 minutes, 10 samples were collected. Sample frequency is 30 seconds (default). This can be set in the <machine_agent_home>/extensions/ServerMonitoring/conf/ServerMonitoring.yml file by changing the "sampling interval" property. Load average is set at 0 before computing. Then, for each sample collected at the sampling frequency, the load average is calculated as an exponentially-damped moving average. The data points are added into the load average results for each sample. exp = e power -(sampleFrequency/interval) loadAverage = loadAverage * exp loadAverage = loadAverage + ProcessorQueueLength*(1-exp)​
Symptoms While taking a metadata backup of the Controller using the ControllerMetadataBackup.sh script (as outlined in the Controller Data Backup and Restore documentation) it fails with error me... See more...
Symptoms While taking a metadata backup of the Controller using the ControllerMetadataBackup.sh script (as outlined in the Controller Data Backup and Restore documentation) it fails with error message: mysqldump: Got error: 2002: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2) when trying to connect  Diagnosis For mysqldump (used in the ControllerMetadataBackup.sh script): If host is set to localhost , then a socket or pipe is used. If host is set to 127.0.0.1 , then the client is forced to use TCP/IP. No host is set in the script by default, therefore on some servers, the script might fail while trying to use socket to connect to MySQL. Solution Add -h  with the Controller hostname to the mysqldump options in the ControllerMetadataBackup.sh script. This forces mysqldump to use TCP/IP. Change from: $appd_install_dir/db/bin/mysqldump -u root -p[password] -P [port_number] controller  To:   $appd_install_dir/db/bin/mysqldump -u root -p[password] -h [controller_host_name] -P [port_number] controller 
Users can redirect requests from HTTP to the HTTPS Controller port on Linux using Iptables. In your computer terminal, replace eth0 with your actual interface name using the following command:... See more...
Users can redirect requests from HTTP to the HTTPS Controller port on Linux using Iptables. In your computer terminal, replace eth0 with your actual interface name using the following command: sudo iptables -A PREROUTING -t nat -i eth0 -p tcp --dport <HTTP_PORT> -j REDIRECT --to-port <HTTPS_PORT> If port 8090 and 8181 are not open, run the following command: sudo iptables -A INPUT -i eth0 -p tcp --dport <HTTP_PORT> -j ACCEPT Example: Note: Assuming interface "eth0," HTTP port "8090," and HTTPS port "8181." sudo iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 8090 -j REDIRECT --to-port 8181 sudo iptables -A INPUT -i eth0 -p tcp --dport 8090 -j ACCEPT