How do I set up APM for Events Service when deploying Analytics on premises?

[#1]

What are the steps to set up and configure APM for Events Service, when deploying Analytics on-prem?

Overview

The Events Service should have its own Java Agent and Machine Agent.

• Step 1. How do I set up Machine Agents for Events Service nodes?
• Step 2. Set up Application Agents for the Events Service
• Step 3. Set Up an Application Agent for Elasticsearch
• Step 4. Begin Monitoring
• Troubleshooting
  • Where do I view logs? 
  • How do I troubleshoot common problems? 

Step 1. How do I set up Machine Agents for Events Service Nodes?

To monitor the hardware health of the Events Service nodes, this example uses a Machine Agent installation. For more details, see the AppDynamics Machine Agent installation documentation.

Link each Machine Agent with either:

• the Events Service's API,  or
• the Events Service's Elasticsearch Java Agents

The unique host ID configuration option enables the linking.

Complete the following procedure:

1. Decide whether you are linking the Machine Agent to the Events Service API or the Events Service Elasticsearch.
   
   
2. Note the relevant uniqueHostId value that you configured in the vmoptions file. For example: 

-
Dappdynamics.agent.uniqueHostId=EventsService01

3. Edit <machine-agent-home>\conf\controller-info.xml such that <unique-host-id>EventsService01</unique-host-id> corresponds to that value.
   
   
4. Verify that the Java Agent and Machine Agent are associated by viewing the Applications > Tiers and Nodes tab in the Controller:

Step 2. Set up application Agents for the Events Service

Common Application Agent Configurations

1. Locate your Events Service and Java Agent directories. For this example, use the following:
   /opt/appdynamics/events-service = <Events-Service-Home>
/opt/appdynamics/AppAgent = <AppAgent-Home>
   
   
2. Collect Controller information by using the information in the following table:
   

   Item	Example Data
   hostname	<controller_host>
   port	<controller_port>
   account name	<account_name>
   access key	<access_key>
   name of the APM application that monitors the Events Service	EventsService

3. Edit the controller-info.xml file.
   Add your equivalent to <controller_host>, <controller_port>, <account_name>, <access_key> to AppAgent <AppAgent-Home>/conf/ or the relative directory to be <AppAgent-Home>/conf/controller-info.xml
   
   
4. Edit the events-service.vmoptions file under <Events-Service-Home>/processor/config>

1. Add -javaagent:<AppAgent-Home>/javaagent.jar in the events-service.vmoptions file
2. Stop the event service.
   ./bin/events-service.sh stop
3. Restart the event service to match the new configuration. 
   ./bin/events-service.sh start -p ./conf/events-service-api-store.properties &

Step 3. Set Up an Application Agent for Elasticsearch

1. Add a security policy for AppAgent file to access from within the Elasticsearch process. Create the <Events-Service-Home>/conf/elasticsearch.policy file:

grant codeBase "file:<AppAgent-Home>/-”
{
Permission java.security.AllPermission;
};

2. Create and edit a new <Events-Service-Home>/conf/elasticsearch.vmoptions file.
1. Copy the existing <Events-Service-Home>/conf/events-service.vmoptions file
2. Enable a separate vmoptions file for Elasticsearch by editing the <Events-Service-Home>/conf/events-service-api-store.properties property as follows:
   ad.es.jvm.options.name=elasticsearch.vmoptions
   
   
3. Append the following new properties to the new elasticsearch.vmoptions file.

-javaagent:<AppAgent-Home>/javaagent.jar
-Dappdynamics.socket.collection.bci.enable=true
-Dappdynamics.agent.tierName=EventsService-Elasticsearch
-Dappdynamics.agent.nodeName=Elasticsearch01
-Djava.security.policy=<Events-Service-Home>/conf/elasticsearch.policy

4. Specify a hard (not relative) path in -Djava.security.policy.
   
   
5. Append the following new properties to events-service.vmoptions

-javaagent:<AppAgent-Home>/javaagent.jar
-Dappdynamics.socket.collection.bci.enable=true
-Dappdynamics.agent.uniqueHostId=EventsService01
-Dappdynamics.agent.tierName=EventsService-API
-Dappdynamics.agent.nodeName=API01

Step 4. Begin Monitoring

1. Manually restart the single Events Service process that you are trying to instrument:
   

   bin/events-service.sh stop
   bin/events-service.sh start -p conf/events-service-api-store.properties &

   
   
2. Through the Controller UI, verify that both Application Agents are reporting:
   
   
   
3. To prevent termination after logoff of jobs, detach the job that you just started from the active terminal by using the disown command:
   

   #[1] bin/events-service.sh start -p conf/events-service-api-store.properties disown %1

Troubleshooting

• Where do I view logs?
• How do I troubleshoot common problems?

Where do I view logs?

View the logs from their typical locations, shown in the following table:

Log	Location
API store	<Events-Service-Home>/logs/events-service-api-store.log
Elasticsearch	<Events-Service-Home>/logs/appdynamics-events-service-cluster.log
Java Application Agent	<AppAgent-Home>/ver<Your App Agent Version>/logs/<Your agent.nodeName>/agent<Latest App Agent Start Time>.log

How do I troubleshoot common problems?

Run troubleshooting commands when you encounter common problems.

Problem	Command
When Elasticsearch is starting, you see vm.max_map_count errors	sudo sysctl -w vm.max_map_count=262144
If the system starts running out of descriptors, check /etc/security/limits.conf	Substitute the name of the user running Elasticsearch for the asterisk ("*") echo "*" soft nofile   65536" | sudo tee -a etc/security/limits.conf echo "*" hard nofile   65536" | sudo tee -a etc/security/limits.conf