Event Service RAM requirements, heap memory configuration, and how to trouble shoot the performance-related issues
Introduction
When it comes to an Event Service deployment it is crucial to ensure that there is sufficient RAM available on the OS, and the heap settings are configured correctly on each of the nodes in the cluster. This article will help you understand how to determine how much RAM the Event Service processes are requiring, how to change the heap settings appropriately, and how to trouble shoot memory related issues.
Table of contents
- Determine how much RAM is needed for the environment
- Configure the HEAP memory appropriately depending on your environment
- Tune the Operating System for Production Cluster Nodes
- POV/Testing environments
- Issues with Event Service Performance, and how to trouble shoot them
Determine how much RAM is needed for the environment
As per the AppDynamics official recommendations, for heap space allocation, AppDynamics recommends allocating half of the available RAM to the Events Service process, with a minimum of 7 GB up to 31 GB. Heap memory configuration refers to the settings that determine the amount of memory allocated to the heap in a Java Virtual Machine (JVM). The heap is the runtime data area from which memory for all class instances and arrays is allocated. By carefully configuring these parameters, you can optimize the performance of Java applications and reduce the likelihood of memory-related issues.
Keep in mind that in case you have an environment with plenty of Analytics reporting to your Event Service, the minimum requirement of 7 GB might not be enough. The requirement is based on the amount of load from both Transaction Analytics and Log Analytics events, and depends highly on the use-case and the amount of load in your clsuter. The optimal setup for a production environment is to have:
- At least 62 GB of RAM on the system/host machine,
- At least three nodes.
Everything below that should be used only for POV and testing purposes.
To check the exact requirements based on the License Units consumed in your environment, please refer to the below table:
Event Type Machine Instancei2.2xlarge (61 GB RAM, 8 vCPU, 1600 GB SSD) i2.4xlarge (122 GB RAM, 16 vCPU, 3200 GB SSD) i2.8xlarge (244 GB RAM, 32 vCPU, 6400 GB SSD)1 node 3 nodes 5 nodes 7 nodes 1 node 3 nodes 5 nodes 7 nodes 1 node 3 nodes 5 nodes|
Transaction Analytics license units |
20 | 37 | 44 | 63 | 22 | 41 | 84 | 113 | 53 | 94 | 120 |
| Log Analytics license units | 7 | 10 | 17 | 19 | 16 | 19 | 32 | 44 | 39 | 116 | 270 |
Configure the HEAP memory appropriately depending on your environment
Once you have deployed the appropriate amount of RAM on the system/host machine, it is essential to configure the heap memory settings inside the .properties file of the Event Service appropriately. The heap memory needs to be adjusted on all of the nodes in your cluster. The properties file can be found in the following path:
appdynamics/platform/product/events-service/processor/conf/events-service-api-store.properties
Inside the events-service-api-store.properties we have four settings responsible for HEAP memory allocation:
ad.jvm.heap.min=2048m
ad.jvm.heap.max=2048m
ad.es.jvm.heap.min=4096mm
ad.es.jvm.heap.max=4096mm
The overall recommendation:
Assign the half of the available RAM to the Events Service process. The half of the available RAM should be assigned to ad.jvm.heap and ad.es.jvm.heap settings in 1:3 proportions.
For example: if you have 62GB of RAM on the system/host machine, first it needs to be divided by 2.
62 / 2 = 31 GB of RAM
The result needs to be assigned to ad.jvm.heap and ad.es.jvm.heap settings in 1:3 proportions. In this particular example you need to assign 31 GB of RAM in 1:3 proportions.
Because of this condition, the best way to proceed further is to divide 31 GB by 4. The result 7.75 GB (7750m) needs to be assigned to ad.jvm.heap. Next, multiply this value by 3 (7750 * 3 = 23250) and assign this value to ad.es.jvm.heap.
The heap memory configuration for 62 GB of RAM on the system/host machine should look as follows:
ad.jvm.heap.min=7750m
ad.jvm.heap.max=7750m
ad.es.jvm.heap.min=23250m
ad.es.jvm.heap.max=23250m
After doing any changes inside the .properties file, it is extremely important to restart your Event Service for those changes to be applied.
Tune the Operating System for Production Cluster Nodes
A crucial step which is often skipped during the Event Service deployment process is the procedure of tuning the Operating System for Production Cluster Nodes.
Before installing the Events Service cluster, you need to perform a few manual changes as described below. These are particularly relevant for production Events Service deployments. On each node in the cluster, make these configuration changes:
1. Using a text editor, open /etc/sysctl.conf and add the following:
vm.max_map_count=262144
2. Raise the open file descriptor limit in /etc/security/limits.conf, as follows:
<username_running_eventsservice> soft nofile 96000
<username_running_eventsservice> hard nofile 96000
<username_running_eventsservice> soft memlock unlimited
<username_running_eventsservice> hard memlock unlimited
3. Disable swap memory by running the following command. Remove swap mount points by removing or commenting the lines in /etc/fstab that contain the word swap.
swapoff -a
POV/Testing environments
For a POV/Testing environment, there is no need to have three nodes and 62 GB of RAM available on the node. To configure the Event Service as a single-node deployment, please refer to the following article:
https://community.appdynamics.com/t5/Share-a-tip/Run-Event-Service-as-a-single-node-cluster/td-p/581...
When it comes to the RAM requirements on the system/host machine for a POV/Testing environment, it highly depends on the amount of load from both Transaction Analytics and Log Analytics events you are planning on using in this environment. It is still recommended to have around 20 to 32 GB on the system/host machine, even for testing environments. Anything below that might not be sufficient. Keep in mind that based on the amount of load on the Event Service cluster, this recommendation might not be enough, even in the case of a POV/Testing environment. If you run into any performance issues, please refer to steps n.1, n.2, n.3, n.4 and n.5 in the below "Issues with Event Service Performance" section to determine how much RAM your Elasticsearch processes are consuming.
Issues with Event Service Performance, and how to trouble shoot them
In case you are experiencing a situation where your Event Service is experiencing performance issues or it crashes by itself after running successfully for couple of hours/couple of days, you might have ran into the memory consumption issue. Such situations are almost always related to not enough resources available on your OS, or insufficient heap memory configuration on your Event Service.
To check on this issue, and understand how much RAM your Elastic Search process is consuming, kindly please follow can use the following trouble shooting steps.
1. First, start your Event Service normally (if it is not already running), and make sure it is running. To check that from the CLI level, you can use the following steps methods:
- Check the ps -ef | grep java. You should see two processes running on your OS - one is the Event Service process, another one is the elasticsearch process.
- Check the content of /processor directory - you should see events-service-api-store.id and elasticsearch.id files inside
- Check if the processes are running on appropriate ports. You can check that using netstat -tulpn
Keep in mind, that if your RAM memory is not sufficient, or the HEAP memory configuration is too low your Event Service might not start at all in the first place. In such case, please re-visit the above sections on how to determine how much RAM is needed for the environment, and how to configure the HEAP memory appropriately depending on your environment. Ensure that you have appropriate amount of RAM on your OS, and the heap memory configuration is appropriate.
2. Check and note down the process ID that is associated to the Elastic Search process. If your Elastic Search is running on port 9200 (by default settings), you can use the below output for a reference:
[appdynamics@krk-vnap-461-146-12 processor]$ netstat -anp |grep 9200 |grep LIST
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 :::9200 :::* LISTEN 495838/java
[appdynamics@krk-vnap-461-146-12 processor]$
In the above environment the Elastic Search process ID is 495838. Make sure you node down this process ID.
3. Check the memory consumption of the process:
On Linux systems, use the following command to check how much RAM the Elastic Search process is utilizing at the given moment of time:
echo 0 $(awk '/Rss/ {print "+", $2}' /proc/<process-id-noted-in-step-n.2>/smaps) | bc
The above command returns the memory consumption in KB. Compare this value agains the heap memory and the RAM memory available on your OS. If the memory used is at the edge of the heap limit config or the RAM available, it indicates that the process requires additional RAM memory to run smoothly.
4. In case the Operating System terminates your Elasticsearch process by itself, there is a high possibility it was terminated due to Out of Memory condition. You can check the system-logs to understand why it had happened in the first place. Once you notice your Event Service crashed, check and confirm if the Elastic Search process got terminated:
[appdynamics@krk-vnap-461-146-12 processor]$ netstat -anp |grep 9200 |grep LIST
[appdynamics@krk-vnap-461-146-12 processor]$
As you can see from the above outputs, the Elastic Search process is no longer running,
5. Check the system logs to understand why the Elastic Search process was terminated by your operating system. To do that, you can use the dmesg | grep <process-id-noted-in-step-n.2> command. Please refer to the below output for reference:
[appdynamics@krk-vnap-461-146-12 processor]$ dmesg | grep 495838
[1362539.578533] [ 495838] 1010 495838 6399752 4925562 40865792 0 0 java
[1362539.578544] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1010.slice/session-978.scope,task=java,pid=495838,uid=1010
[1362539.578758] Out of memory: Killed process 495838 (java) total-vm:25599008kB, anon-rss:19702248kB, file-rss:0kB, shmem-rss:0kB, UID:1010 pgtables:39908kB oom_score_adj:0
[appdynamics@krk-vnap-461-146-12 processor]$
The above outputs indicate that the AppDynamics Event Service running on the system is being terminated due to an "Out of Memory" (OOM) condition. The oom-kill message indicates that the system's Out of Memory (OOM) Killer was triggered. This happens when the system runs out of available memory, or the process heap memory configuration was not sufficient. In such case, the system starts terminating processes to free up resources.
From the above snipped, we can conclude that the process was using approximately 19,702,248 kB (about 19.7 GB) of anonymous resident set size (anon-rss), which is the actual memory the process was consuming. Due to insufficient heap memory configuration, the process was terminated.
If you face the above scenarios, it is clear that your Event Service does not have sufficient RAM available to run smoothly. If you encounter such issues, please re-visit the above sections on how to determine how much RAM is needed for the environment, how to tune the Operating System for Production Cluster Nodes and how to configure the HEAP memory appropriately depending on your environment.
If you encounter any further issues with your Event Service component, please consider raising a Support ticket for further help.
