SNI checks An SNI (Server Name Indication) check is a security validation performed during the TLS (SSL) handshake. It verifies that the hostname a client is trying to connect to matches one of the domain names listed in the server's SSL certificate. This mechanism allows a single server to host multiple secure websites on the same IP address, as the client explicitly indicates which website it intends to reach, enabling the server to present the correct certificate. How it works SNI works by having the client include the hostname it is trying to connect to at the very beginning of the TLS handshake process. Client Initiates TLS Handshake, ClientHello Message with SNI. As mentioned above, as part of the initial "ClientHello" message, the client includes an SNI extension. This extension contains the domain name (hostname) of the server it intends to reach. Server Selects Certificate. The destinaction server receives the ClientHello message and, using the hostname provided in the SNI field, identifies which of its multiple SSL certificates is appropriate for that specific domain. It then iterates through the certificates avlible in the trustore, and examines their Subject Alternative Name (SAN) field. The SAN field can contain multiple DNS entries (domain names). The server looks for a direct match between the SNI hostname and any of the DNS entries in the certificate's SAN field. Server performs the SNI checks If there is no match between the Hostname received in the ClientHello Message and the certificate SAN list, the server throws SNI Error, If there is a match between the Hostname received in the ClientHello Message and the certificate SAN list, the Server Presents Correct Certificate. The server than sends back the correct SSL certificate to the client. The rest of the TLS handshake proceeds as usual, establishing a secure, encrypted connection. SNI Checks in AppDynamics components The newest version of AppDynamics components usually come bundled with the Java 17 version. OpenJDK 17 and later versions enforce SNI restrictions more strictly than previous Java versions. Java 17+ enforces that SNI hostnames must contain at least one dot, unlike Java 11 (and lower) which was less restrictive. This means that even if Jetty's internal SNI logic hasn't changed drastically, the underlying Java platform's behavior can make SNI checks appear stricter. Upgrading, for example, the AppDynamics Event Service to a version that utilizes Java 17 can lead to Server Name Indication (SNI) checks errors, especially if the previous deployment used Java 8. This issue typically manifests in the logs with entries similar to the following: [ERROR] [quota-cache-refresher-1] [c.a.a.p.event.meter.DefaultMeters] Error attempting to get document fragments used count for account [<account-name>] event type [<event-type>]. Defaulting to [0] com.appdynamics.analytics.client.common.exceptions.BadRequestRestException: Status code: [400], Message: Invalid SNI This problem most commonly happens because the server certificate lacks the appropriate Subject Alternative Name (SAN) domain entries. While the same certificate configuration might have functioned correctly with older Event Service versions and services running Java 11 (or older), AppDynamics 24.10 versions (and newer) releases bundle newer Java versions that enforce stricter SNI validation requirements, causing the previous setup incompatible. Note: SNI will not be sent if the destination host is localhost or an IP address; a hostname is required for the SNI checks to be triggered. This means that if you are using raw IP addresses in your environment, you won't be impacted by an issues with SNI checks. Resolving Issues with AppDynamics SNI checks In order to address this issue, you can either fix the hostnames and SAN list in your certificate configuration, or disable the SNI checks for AppDynamics components. Verifying hostname configurations to ensure proper functionality of SNI (Server Name Indication) checks. If you want to use the SNI check in your environment for security reasons, you need to ensure that the certificate used by the AppDynamics services contains appropriate hostnames in the CN or SAN sections. 1. First of all, check the exact hostnames with domain names used by your servers in the cluster: hostname -f hostname --fqdn   2. Next, check the SAN list in the certificate the server uses for communication: openssl x509 -in certificate.pem -text -noout Look for the "Subject Alternative Name" section within the output presented by the command above.   3. Ensure that the Subject Alternative Name within your certificate has appropraite hostnames with domain names configured. The SAN section of the certificate should contain all of the hostnames.domain you are using in your cluster. Example output: X509v3 Subject Alternative Name: DNS:<controller-hostname>.domain.name, DNS:<eum-hostname>.domain.name DNS:<es-node-1-hostname>.domain.name, DNS:<es-node-2-hostname>.domain.name ,DNS:<es-node-3-hostname>.domain.name Please make sure that the SAN section of your certificate contains the correct hostname, together with the appropriate domain. Java 17+ enforces that SNI hostnames must contain at least one dot, unlike Java 11 which was less restrictive. Make sure that you are using hostnames together with domain names in your environment, in case you want to use the SNI checks appropriately. Disabling the SNI checks. You can simply disable the SNI checks by changing the AppDynamics components configuration. Please remember to take a back up of the configuration files before making any changes. In both on-prem and OVA environments, a service restart is required for configuration changes to take effect. After applying the changes below, restart the environment to ensure the new settings are applied. If you modify the SNI checks setting for the Event Service only, you only need to restart the Event Service component—there is no need to restart the Controller or EUM. The same applies if you change the SNI checks setting exclusively for the Controller or EUM; only the affected component requires a restart. On-prem environments. 1. Event Service You can disable the SNI checks by modifying the  appdynamics/platform/product/events-service/processor/conf/events-service-api-store.yml file, and adding disableSniHostCheck: true under the server -> applicationConnectors section. See this link for more details: https://www.dropwizard.io/en/stable/manual/configuration.html Sample configuration: server: applicationConnectors: - type: https port: ${ad.dw.http.port} disableSniHostCheck: true 2. Controller You can disable the SNI checks by opening the  <controller_home>/appserver/jetty/etc/jetty-ssl.xml with an editor, and updating the sniHostCheck value from "default=true" to "default=false": Before: <Arg name="sniHostCheck" type="boolean"><Property name="jetty.ssl.sniHostCheck" default="true"/></Arg> After: <Arg name="sniHostCheck" type="boolean"><Property name="jetty.ssl.sniHostCheck" default="false"/></Arg>   3.EUM server You can disable the SNI checks by modifying the appdynamics/EUM/eum-processor/bin/eum-processor-launcher.vmoptions file and adding -Djetty.ssl.sniHostCheck=false     OVA environments. 1. Event Service OVA component Navigate to the OVA installation directory for the Event Service templates, and edit the following file with text editor: <OVA installation path>/appd-charts/charts/events/templates/events-service-configmap.yml Add the disableSniHostCheck: true configuration under the server -> applicationConnector sub-section in `events-service-api-store.yml:` section. See this link for more details: https://www.dropwizard.io/en/stable/manual/configuration.html Sample configuration: server: applicationConnectors: - type: https port: ${ad.dw.http.port} disableSniHostCheck: true   2. Controller OVA component Navigate to the OVA installation directory for the Controller templates, and edit the following file with a text editor: <OVA installation path>/appd-charts/charts/controller/templates/controller-configmap.yml Update the sniHostCheck  value from "default=true" to "default=false" under the jetty-ssl.xml section: Before: <Arg name="sniHostCheck" type="boolean"><Property name="jetty.ssl.sniHostCheck" default="true"/></Arg> After: <Arg name="sniHostCheck" type="boolean"><Property name="jetty.ssl.sniHostCheck" default="false"/></Arg>   3. EUM OVA component  Navigate to the OVA installation directory for the EUM templates, and edit the following file with a text editor: <OVA installation path>/appd-charts/charts/eum/templates/eum-configmap.yml Add the -Djetty.ssl.sniHostCheck=false configuration under the eum-processor-launcher.vmoptions section.   In case of any issues, feel free to reach out to the AppDynamics support organization for further support. ... View more

The AppDynamics Private Synthetic Agent relies on Docker images to function as designed. If your environment has access to external networks, you can download the required images directly either from docker.io using the docker pull command or directly from the AppDynamics downloads portal, as per the steps in the official documentation. However, if your environment does not have internet access, you must manually download and upload these images. This process is explained below in details.  1. On a machine that has access to the internet, pull the required images from the docker.io repository using the below commands: docker pull apacheignite/ignite:2.16.0-jdk11 docker pull appdynamics/heimdall-psa docker pull appdynamics/chrome-agent-psa docker pull appdynamics/api-monitoring-agent-psa Note: You can also download the repositories directly from AppDynamics downloads portal, as per the documentation 2. Once downloaded, save each of the repositories to a .tar file:  docker save -o ignite.tar apacheignite/ignite:2.16.0-jdk11 docker save -o heimdall.tar appdynamics/heimdall-psa docker save -o chrome-agent.tar appdynamics/chrome-agent-psa docker save -o api-agent.tar appdynamics/api-monitoring-agent-psa 3. All of the .tar files created using the above commands need to be copied to the destination PSA agent machine.  4. Once all of the .tar files were successfully transferred, deploy the repositories using the below commands:  docker load -i ignite.tar docker load -i heimdall.tar docker load -i chrome-agent.tar docker load -i api-agent.tar 5. Check if the repositories are loaded successfully using docker image ls. The output of the docker image ls should look something similar to this:  docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE appdynamics/chrome-agent-psa latest 2b15215e727c 3 weeks ago 1.63GB appdynamics/api-monitoring-agent-psa latest d2b6157359da 3 weeks ago 138MB appdynamics/heimdall-psa latest a38a9b4a0c81 3 weeks ago 533MB apacheignite/ignite 2.16.0-jdk11 b043b94a51cf 20 months ago 383MB Note: The REPOSITORY:TAG values are often referenced inside the .yaml configuration files used during the deployment process. In case of any issues, kindly please: In case of orchestrated environments (such as Minikube, OpenShift, etc.), ensure that inside the values.yaml file, the repository name and tag for heimdall, chromeAgent, apiMonitoringAgent and ignite is correct, In case of docker deployment, ensure that inside the docker-compose.yaml file, the repository name and tag for heimdall and ignite is correct. In addition to that, ensure that inside the  application-onprem.yml file, the repository name and tag for chrome and api is correct.  For more details, please feel free to go through our Official Documentation. In case you are still facing issues, please raise a Service Request to the AppDynamics support organisation.   ... View more

Introduction When it comes to an Event Service deployment it is crucial to ensure that there is sufficient RAM available on the OS, and the heap settings are configured correctly on each of the nodes in the cluster. This article will help you understand how to determine how much RAM the Event Service processes are requiring, how to change the heap settings appropriately, and how to trouble shoot memory related issues.  Table of contents  Determine how much RAM is needed for the environment  Configure the HEAP memory appropriately depending on your environment Tune the Operating System for Production Cluster Nodes POV/Testing environments Issues with Event Service Performance, and how to trouble shoot them Determine how much RAM is needed for the environment As per the AppDynamics official recommendations, for heap space allocation, AppDynamics recommends allocating half of the available RAM to the Events Service process, with a minimum of 7 GB up to 31 GB. Heap memory configuration refers to the settings that determine the amount of memory allocated to the heap in a Java Virtual Machine (JVM). The heap is the runtime data area from which memory for all class instances and arrays is allocated. By carefully configuring these parameters, you can optimize the performance of Java applications and reduce the likelihood of memory-related issues. Keep in mind that in case you have an environment with plenty of Analytics reporting to your Event Service, the minimum requirement of 7 GB might not be enough. The requirement is based on the amount of load from both Transaction Analytics and Log Analytics events, and depends highly on the use-case and the amount of load in your clsuter. The optimal setup for a production environment is to have: At least 62 GB of RAM on the system/host machine, At least three nodes. Everything below that should be used only for POV and testing purposes. To check the exact requirements based on the License Units consumed in your environment, please refer to the below table: Event Type Machine Instancei2.2xlarge (61 GB RAM, 8 vCPU, 1600 GB SSD) i2.4xlarge (122 GB RAM, 16 vCPU, 3200 GB SSD) i2.8xlarge (244 GB RAM, 32 vCPU, 6400 GB SSD)1 node 3 nodes 5 nodes 7 nodes 1 node 3 nodes 5 nodes 7 nodes 1 node 3 nodes 5 nodes Transaction Analytics license units 20 37 44 63 22 41 84 113 53 94 120 Log Analytics license units 7 10 17 19 16 19 32 44 39 116 270   Reference: https://docs.appdynamics.com/appd/onprem/23.x/23.11/en/events-service-deployment/events-service-requirements#id-.EventsServiceRequirementsv23.9-EventsServiceNodeSizingBasedonLicenseUnits Configure the HEAP memory appropriately depending on your environment Once you have deployed the appropriate amount of RAM on the system/host machine, it is essential to configure the heap memory settings inside the .properties file of the Event Service appropriately. The heap memory needs to be adjusted on all of the nodes in your cluster. The properties file can be found in the following path:     appdynamics/platform/product/events-service/processor/conf/events-service-api-store.properties     Inside the events-service-api-store.properties we have four settings responsible for HEAP memory allocation:   ad.jvm.heap.min=2048m ad.jvm.heap.max=2048m ad.es.jvm.heap.min=4096mm ad.es.jvm.heap.max=4096mm   The overall recommendation: Assign the half of the available RAM to the Events Service process. The half of the available RAM should be assigned to ad.jvm.heap and ad.es.jvm.heap settings in 1:3 proportions. For example: if you have 62GB of RAM on the system/host machine, first it needs to be divided by 2.   62 / 2 = 31 GB of RAM   The result needs to be assigned to ad.jvm.heap and ad.es.jvm.heap settings in 1:3 proportions. In this particular example you need to assign 31 GB of RAM in 1:3 proportions. Because of this condition, the best way to proceed further is to divide 31 GB by 4. The result 7.75 GB (7750m) needs to be assigned to ad.jvm.heap. Next, multiply this value by 3 (7750 * 3 = 23250) and assign this value to ad.es.jvm.heap. The heap memory configuration for 62 GB of RAM on the system/host machine should look as follows:   ad.jvm.heap.min=7750m ad.jvm.heap.max=7750m ad.es.jvm.heap.min=23250m ad.es.jvm.heap.max=23250m   After doing any changes inside the .properties file, it is extremely important to restart your Event Service for those changes to be applied.   Tune the Operating System for Production Cluster Nodes A crucial step which is often skipped during the Event Service deployment process is the procedure of tuning the Operating System for Production Cluster Nodes. Before installing the Events Service cluster, you need to perform a few manual changes as described below. These are particularly relevant for production Events Service deployments. On each node in the cluster, make these configuration changes: 1. Using a text editor, open /etc/sysctl.conf and add the following:   vm.max_map_count=262144   2. Raise the open file descriptor limit in /etc/security/limits.conf, as follows:   <username_running_eventsservice> soft nofile 96000 <username_running_eventsservice> hard nofile 96000 <username_running_eventsservice> soft memlock unlimited <username_running_eventsservice> hard memlock unlimited   3. Disable swap memory by running the following command. Remove swap mount points by removing or commenting the lines in /etc/fstab that contain the word swap.   swapoff -a   Reference: https://docs.appdynamics.com/appd/onprem/24.x/latest/en/events-service-deployment/install-the-events-service-on-linux#id-.InstalltheEventsServiceonLinuxv24.4-TunetheOperatingSystemforProductionClusterNodes   POV/Testing environments  For a POV/Testing environment, there is no need to have three nodes and 62 GB of RAM available on the node. To configure the Event Service as a single-node deployment, please refer to the following article: https://community.appdynamics.com/t5/Share-a-tip/Run-Event-Service-as-a-single-node-cluster/td-p/58121 When it comes to the RAM requirements on the system/host machine for a POV/Testing environment, it highly depends on the amount of load from both Transaction Analytics and Log Analytics events you are planning on using in this environment. It is still recommended to have around 20 to 32 GB on the system/host machine, even for testing environments. Anything below that might not be sufficient. Keep in mind that based on the amount of load on the Event Service cluster, this recommendation might not be enough, even in the case of a POV/Testing environment. If you run into any performance issues, please refer to steps n.1, n.2, n.3, n.4 and n.5 in the below "Issues with Event Service Performance" section to determine how much RAM your Elasticsearch processes are consuming.   Issues with Event Service Performance, and how to trouble shoot them In case you are experiencing a situation where your Event Service is experiencing performance issues or it crashes by itself after running successfully for couple of hours/couple of days, you might have ran into the memory consumption issue. Such situations are almost always related to not enough resources available on your OS, or insufficient heap memory configuration on your Event Service. To check on this issue, and understand how much RAM your Elastic Search process is consuming, kindly please follow can use the following trouble shooting steps. 1. First, start your Event Service normally (if it is not already running), and make sure it is running. To check that from the CLI level, you can use the following steps methods: Check the ps -ef | grep java. You should see two processes running on your OS - one is the Event Service process, another one is the elasticsearch process. Check the content of /processor directory - you should see events-service-api-store.id and elasticsearch.id files inside Check if the processes are running on appropriate ports. You can check that using netstat -tulpn Keep in mind, that if your RAM memory is not sufficient, or the HEAP memory configuration is too low your Event Service might not start at all in the first place. In such case, please re-visit the above sections on how to determine how much RAM is needed for the environment, and how to configure the HEAP memory appropriately depending on your environment. Ensure that you have appropriate amount of RAM on your OS, and the heap memory configuration is appropriate.   2. Check and note down the process ID that is associated to the Elastic Search process. If your Elastic Search is running on port 9200 (by default settings), you can use the below output for a reference:   [appdynamics@krk-vnap-461-146-12 processor]$ netstat -anp |grep 9200 |grep LIST (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp6 0 0 :::9200 :::* LISTEN 495838/java [appdynamics@krk-vnap-461-146-12 processor]$   In the above environment the Elastic Search process ID is 495838. Make sure you node down this process ID.   3. Check the memory consumption of the process: On Linux systems, use the following command to check how much RAM the Elastic Search process is utilizing at the given moment of time:   echo 0 $(awk '/Rss/ {print "+", $2}' /proc/<process-id-noted-in-step-n.2>/smaps) | bc   The above command returns the memory consumption in KB. Compare this value agains the heap memory and the RAM memory available on your OS. If the memory used is at the edge of the heap limit config or the RAM available, it indicates that the process requires additional RAM memory to run smoothly.   4. In case the Operating System terminates your Elasticsearch process by itself, there is a high possibility it was terminated due to Out of Memory condition. You can check the system-logs to understand why it had happened in the first place. Once you notice your Event Service crashed, check and confirm if the Elastic Search process got terminated:   [appdynamics@krk-vnap-461-146-12 processor]$ netstat -anp |grep 9200 |grep LIST [appdynamics@krk-vnap-461-146-12 processor]$   As you can see from the above outputs, the Elastic Search process is no longer running, 5. Check the system logs to understand why the Elastic Search process was terminated by your operating system. To do that, you can use the dmesg | grep <process-id-noted-in-step-n.2> command. Please refer to the below output for reference:   [appdynamics@krk-vnap-461-146-12 processor]$ dmesg | grep 495838 [1362539.578533] [ 495838] 1010 495838 6399752 4925562 40865792 0 0 java [1362539.578544] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1010.slice/session-978.scope,task=java,pid=495838,uid=1010 [1362539.578758] Out of memory: Killed process 495838 (java) total-vm:25599008kB, anon-rss:19702248kB, file-rss:0kB, shmem-rss:0kB, UID:1010 pgtables:39908kB oom_score_adj:0 [appdynamics@krk-vnap-461-146-12 processor]$   The above outputs indicate that the AppDynamics Event Service running on the system is being terminated due to an "Out of Memory" (OOM) condition. The oom-kill message indicates that the system's Out of Memory (OOM) Killer was triggered. This happens when the system runs out of available memory, or the process heap memory configuration was not sufficient. In such case, the system starts terminating processes to free up resources. From the above snipped, we can conclude that the process was using approximately 19,702,248 kB (about 19.7 GB) of anonymous resident set size (anon-rss), which is the actual memory the process was consuming. Due to insufficient heap memory configuration, the process was terminated. If you face the above scenarios, it is clear that your Event Service does not have sufficient RAM available to run smoothly. If you encounter such issues, please re-visit the above sections on how to determine how much RAM is needed for the environment, how to tune the Operating System for Production Cluster Nodes and how to configure the HEAP memory appropriately depending on your environment. If you encounter any further issues with your Event Service component, please consider raising a Support ticket for further help. ... View more

Guide for Index Rollovers in AppDynamics Event Service For event service deployments the index will rollover either when the average shard size breaches a threshold or when the age of the index exceeds the data retention period for the account/event type. Regarding the newest releases of Event Service, it is also quite a common scenario in case of recent environment migration. After such migrations, indexes sometimes don't roll over automatically. This issue occurs if data older than the destination Event Service's retention period was migrated. In this particular case, the migrated data is beyond the retention period, so the new Event Service don't include it in the roll-over process. In those cases, you can follow the instructions below to roll it over. The format of the curl request for index roll-over is a bit different for ES8 when compared to legacy ES2.  Before rolling over each index, please make sure that your cluster is in green status. Before and after rolling over an index, execute the following curl and ensure that the cluster is in "green" status, with no shards in unassigned status: curl -s 'http://localhost:9200/_cat/health?v' The template for index roll-over curl command is as follows: curl -XPOST http://{host}:{port}/v1/admin/cluster/{cluster}/index/{index}/rollover -H"Authorization: Basic {key}" -H"Content-Type: application/json" -H"Accept: application/json" -d '{"numberOfShards": "2"}' You need to fill up the following values manually in the above curl command: - {host} - change with hostname, if you are running from the Event Service CLI, this can be set to "localhost", - {port} - change to port event service is bined. By default 9080, refer to .properties file using "grep ad.dw.http.port events-service-api-store.properties", - {cluster} - cluster name, this can be get from .properties file using "grep ad.es.cluster.name events-service-api-store.properties", - {index} - Index to roll-over, the list of indexes cen be checked using the following curl http://localhost:9200/_cat/indices?v - {key} - Base64 encoded ad.accountmanager.key.ops from events-service-api-store.properties file. On linux CLI, to get this value you can for example use: "echo -n {ad.accountmanager.key.ops} | base64" ... View more

Hello,  Per the official AppDynamics documentation, a single node Event Service cluster is not supported for production and should be used for PoV or testing purposes only. By default Event Service will be installed as "production" deployment, and expects such deployment to be on, hence it will fail/crash if it is run on single node. To run a Single node, you will need to configure the events-service-api-store.yml accordingly: Comment out the following line (by putting "#" in front of the below line): cluster.initial_master_nodes:   $ { ad.es.cluster.initial_master_nodes } Add the following line under the above: discovery.type:   single-node Comment out the following line  (by putting "#" in front of the below line): discovery.seed_hosts: ${ad.es.node.unicast.hosts} Restart Event Service to apply new configs. Reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html#single-node-discovery If you encounter any issues when running the above configurations, please reach out to the AppDynamics support organization. ... View more