AppDynamics Knowledge Base
×

Are you a member of the Splunk Community?

Sign in or Register with your Splunk account to get your questions answered, access valuable resources and connect with experts!
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
New Article

How to have two Cluster Agent auto-instrument and report to two different Controllers

How to have two Cluster Agent auto-instrument and report to two different Controllers

There will be times when multiple teams in your Organization will want to monitor and instrument their Applications. They would also want to have these applications report to different AppDynamics Backend (Controller)

The latest Cluster agent releases gives you that option. Below has been tested on Cluster Agent 25.5 release.

 First create 2 Applications:

1. tomcat-sample.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-app
  labels:
    app: tomcat-app-java-apps
  namespace: java-apps
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tomcat-app
  template:
    metadata:
      labels:
        app: tomcat-app
    spec:
      containers:
      - name: tomcat-app
        #image: docker.io/abhimanyubajaj98/tomcat-sample:latest
        image: docker.io/abhimanyubajaj98/tomcat-sample
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        env:
        - name: JAVA_TOOL_OPTIONS
          value: "-Xmx512m"

---
apiVersion: v1
kind: Service
metadata:
  name: tomcat-app-service
  labels:
    app: tomcat-app
  namespace: java-apps
spec:
  ports:
  - port: 8080
    targetPort: 8080
  selector:
    app: tomcat-app

 

2. tomcat-sample-ces.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-app-1
  labels:
    app: tomcat-app-java-apps-1
  namespace: one-apps
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tomcat-app-1
  template:
    metadata:
      labels:
        app: tomcat-app-1
    spec:
      containers:
      - name: tomcat-app-1
        #image: docker.io/abhimanyubajaj98/tomcat-sample:latest
        image: docker.io/abhimanyubajaj98/tomcat-sample
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        env:
        - name: JAVA_TOOL_OPTIONS
          value: "-Xmx512m"

---
apiVersion: v1
kind: Service
metadata:
  name: tomcat-app-service-1
  labels:
    app: tomcat-app-1
  namespace: one-apps
spec:
  ports:
  - port: 8080
    targetPort: 8080
  selector:
    app: tomcat-app-1

 

To deploy->

1. kubectl create ns java-apps one-apps

2. kubectl create -f tomcat-sample.yaml

3. kubectl create -f tomcat-sample-ces.yaml

 

Once deployed:

root@ip-172-31-12-116:/opt/appdynamics/java-apps# kubectl -n java-apps get pods
NAME                          READY   STATUS    RESTARTS   AGE
tomcat-app-76df69dc7b-f74wv   1/1     Running   0          71m
root@ip-172-31-12-116:/opt/appdynamics/java-apps# kubectl -n one-apps get pods
NAME                            READY   STATUS    RESTARTS   AGE
tomcat-app-1-6b86c4b444-h65sj   1/1     Running   0          20m

 

Auto-instrument

For this I will deploy one cluster agent with Helm charts and the other one with normal kubectl command line.

First up is creating the secret. Remember Cluster agent will be deployed in 2 namespaces so we will need to create 2 secrets in different namespace. 

Our 1st namespace  : appdynamics

Our 2nd namespace : appdynamics-1

Let's create secret:

kubectl -n appdynamics create secret generic cluster-agent-secret --from-literal=controller-key=<access-key>

kubectl -n appdynamics-1 create secret generic cluster-agent-secret --from-literal=controller-key=<access-key>

 

First Cluster Agent deployment in appdynamics namespace

My helm:

root@ip-172-31-12-116:/opt/appdynamics/cluster-agent/cluster-agent-alpine-arm64-bundled-distribution/helm-charts# cat values.yaml
installClusterAgent: true
installInfraViz: false
installSplunkOtelCollector: false
imageInfo:
  agentImage: docker.io/appdynamics/cluster-agent
  agentTag: 25.5.0-1126
  operatorImage: docker.io/appdynamics/cluster-agent-operator
  operatorTag: 25.5.0-1107
  imagePullPolicy: Always
  machineAgentImage: docker.io/appdynamics/machine-agent
  machineAgentTag: latest
  machineAgentWinImage: docker.io/appdynamics/machine-agent-analytics
  machineAgentWinTag: win-latest
  netVizImage: docker.io/appdynamics/machine-agent-netviz
  netvizTag: latest
controllerInfo:
  url: https://controllerces.saas.appdynamics.com:443
  account: controllerces
  username: null
  password: null
  accessKey: null
  globalAccount: controllerxxx_xxxxxx3
  customSSLCert: null
  keyStorePasswordSecret: ''
  keyStoreFileSecret: ''
  authenticateProxy: false
  proxyUrl: null
  proxyUser: null
  proxyPassword: null
createServiceAccount: true

clusterAgent:
  containerProperties:
    containerBatchSize: 5
    containerParallelRequestLimit: 1
    containerRegistrationInterval: 120
  logProperties:
    logFileSizeMb: 5
    logFileBackups: 3
    logLevel: DEBUG
  metricProperties:
    metricsSyncInterval: 30
    metricUploadRetryCount: 2
    metricUploadRetryIntervalMilliSeconds: 5
    podMetricCollectionMaxGoRoutines: 3
    podMetricCollectionRequestTimeoutSeconds: 5
  nsToMonitorRegex: .*
  appName: two-ca

instrumentationConfig:
  enabled: true
  containerAppCorrelationMethod: proxy
  instrumentationMethod: Env
  numberOfTaskWorkers: 5
  appNameStrategy: label
  nsToInstrumentRegex: java-apps|nodejs-apps|dotnet-apps
  instrumentationRules:
  - namespaceRegex: java-apps
    language: java
    appNameLabel: app
    runAsUser: 999
    runAsGroup: 999
    imageInfo:
      image: docker.io/appdynamics/java-agent:latest
      agentMountPath: /opt/appdynamics
      imagePullPolicy: Always

 

helm install -f values.yaml cluster-agent -n appdynamcis 

root@ip-172-31-12-116:/opt/appdynamics/cluster-agent/cluster-agent-alpine-arm64-bundled-distribution/helm-charts# kubectl -n appdynamics get pods
NAME                                    READY   STATUS    RESTARTS   AGE
appdynamics-operator-769dcf8f4b-kjm2z   1/1     Running   0          77m
two-ca-appdynamics-54bc66f55c-5qzfk     1/1     Running   0          77m

 

Second Cluster Agent deployment in appdynamics namespace

Before we deploy the second Cluster Agent, we will need to edit our cluster-agent-operator.yaml file.

What we need to make sure is every place where you have 

namespace: appdynamics

This gets changed to

namespace: appdynamics-1

 

I have uploaded cluster-agent-operator.txt file that has this configuration. You can use with 25.5 version of Cluster Agent by changing the extension to yaml

Do->

kubectl create -f cluster-agent-operator.yaml

My cluster-agent.yaml:

root@ip-172-31-12-116:/opt/appdynamics/cluster-agent/cluster-agent-alpine-arm64-bundled-distribution# cat cluster-agent.yaml
apiVersion: cluster.appdynamics.com/v1alpha1
kind: Clusteragent
metadata:
  name: k8s-cluster-agent
  namespace: appdynamics-1
spec:
  appName: "cluster-2"
  controllerUrl: "https://ces-controller.saas.appdynamics.com:443"
  account: "ces-controller"
  # docker image info
  image: "docker.io/appdynamics/cluster-agent:latest"
  serviceAccountName: appdynamics-cluster-agent
  nsToMonitorRegex: appdynamics
  resources:
   limits:
    cpu: 90m
    memory: "200Mi"
   requests:
    cpu: 90m
    memory: "100Mi"
  instrumentationMethod: Env
  nsToInstrumentRegex: one-apps
  appNameStrategy: label
  instrumentationRules:
  - namespaceRegex: one-apps
    language: java
    appNameLabel: app
    runAsUser: 999
    runAsGroup: 999
    imageInfo:
      image: docker.io/appdynamics/java-agent:latest
      agentMountPath: /opt/appdynamics
      imagePullPolicy: Always
  ### Uncomment the following line if you need pull secret
  #imagePullSecret: "<your-docker-pull-secret-name>"
  ### Uncomment the following line if you need to enable the profiling
  #pprofEnabled: true
  #pprofPort: 9991

 

kubectl create -f cluster-agent.yaml

Once done:

root@ip-172-31-12-116:/opt/appdynamics/cluster-agent/cluster-agent-alpine-arm64-bundled-distribution# kubectl -n appdynamics-1 get pods
NAME                                    READY   STATUS    RESTARTS   AGE
appdynamics-operator-7677764d7c-52nx9   1/1     Running   0          51m
k8s-cluster-agent-6dbdb6b4dc-dkdgs      1/1     Running   0          30m

 

Now, what should happen is namespace one-apps should be instrumented with second Cluster Agent aka one running in appdynamics-1 namespace and java-apps should be instrumented with first Cluster Agent aka one running in appdynamics namespace

And that is what happened->

root@ip-172-31-12-116:/opt/appdynamics/cluster-agent/cluster-agent-alpine-arm64-bundled-distribution# kubectl -n java-apps describe pod
Name:             tomcat-app-76df69dc7b-f74wv
Namespace:        java-apps
Priority:         0
Service Account:  default
Node:             ip-172-31-12-116/172.31.12.116
Start Time:       Mon, 30 Jun 2025 19:07:11 +0000
Labels:           app=tomcat-app
                  pod-template-hash=76df69dc7b
Annotations:      APPD_DEPLOYMENT_NAME: tomcat-app
                  APPD_INSTRUMENTED_CONTAINERS: tomcat-app
                  APPD_POD_INSTRUMENTATION_STATE: Successful
                  APPD_tomcat-app_APPNAME: tomcat-app-java-apps
                  APPD_tomcat-app_NODEID: 1588804
                  APPD_tomcat-app_NODENAME: tomcat-app--12
                  APPD_tomcat-app_TIERID: 30171
                  APPD_tomcat-app_TIERNAME: tomcat-app
                  cni.projectcalico.org/podIP: 10.244.116.136/32
                  cni.projectcalico.org/podIPs: 10.244.116.136/32
Status:           Running
IP:               10.244.116.136
IPs:
  IP:           10.244.116.136
Controlled By:  ReplicaSet/tomcat-app-76df69dc7b
Init Containers:
  appd-agent-attach-java:
    Container ID:  containerd://2785c3422edd754d96270af5e970312791079bb6dc9840acd3ab4af84e5985a0
    Image:         docker.io/appdynamics/java-agent:latest
    Image ID:      docker.io/appdynamics/java-agent@sha256:d237aeb95a7b77d6e3e5b2c868e03cd22077e24424b58fa2380e6de340305e35
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
      cp -r /opt/appdynamics/. /opt/appdynamics-java && chown -R 999:999 /opt/appdynamics-java ; ls -la /opt/appdynamics-java
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 30 Jun 2025 19:07:15 +0000
      Finished:     Mon, 30 Jun 2025 19:07:20 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     20m
      memory:  75M
    Requests:
      cpu:        10m
      memory:     50M
    Environment:  <none>
    Mounts:
      /opt/appdynamics-java from appd-agent-repo-java (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-95g58 (ro)
Containers:
  tomcat-app:
    Container ID:   containerd://5a27c1bb0d2f539f8288fc65a99cebabeed69d10d34de85c50d0f1389c4e5701
    Image:          docker.io/abhimanyubajaj98/tomcat-sample
    Image ID:       docker.io/abhimanyubajaj98/tomcat-sample@sha256:19558c877ec1fca506c3b7a6696a7a3a8914e855df3af93778323fb909b2fdff
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Mon, 30 Jun 2025 19:07:20 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      APPDYNAMICS_AGENT_ACCOUNT_ACCESS_KEY:           <set to the key 'controller-key' in secret 'cluster-agent-secret'>  Optional: false
      JAVA_TOOL_OPTIONS:                              -Xmx512m -Dappdynamics.agent.accountAccessKey=$(APPDYNAMICS_AGENT_ACCOUNT_ACCESS_KEY) -Dappdynamics.socket.collection.bci.enable=true -Dappdynamics.jvm.shutdown.mark.node.as.historical=false  -Dappdynamics.agent.reuse.nodeName=true -javaagent:/opt/appdynamics-java/javaagent.jar
      APPDYNAMICS_CONTROLLER_HOST_NAME:               controllerces.saas.appdynamics.com
      APPDYNAMICS_CONTROLLER_PORT:                    443
      APPDYNAMICS_AGENT_TIER_NAME:                    tomcat-app
      APPDYNAMICS_POD_NAMESPACE:                      java-apps
      APPDYNAMICS_CONTAINER_NAME:                     tomcat-app
      APPDYNAMICS_CONTROLLER_SSL_ENABLED:             true
      APPDYNAMICS_AGENT_ACCOUNT_NAME:                 controllerces
      APPDYNAMICS_AGENT_APPLICATION_NAME:             tomcat-app-java-apps
      APPDYNAMICS_JAVA_AGENT_REUSE_NODE_NAME_PREFIX:  tomcat-app
      APPDYNAMICS_CONTAINERINFO_FETCH_SERVICE:        cluster-metadata-service.appdynamics:9090
      APPDYNAMICS_NETVIZ_AGENT_HOST:                   (v1:status.hostIP)
      APPDYNAMICS_NETVIZ_AGENT_PORT:                  3892
    Mounts:
      /opt/appdynamics-java from appd-agent-repo-java (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-95g58 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  appd-agent-repo-java:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-95g58:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

 

Name:             tomcat-app-1-6b86c4b444-h65sj
Namespace:        one-apps
Priority:         0
Service Account:  default
Node:             ip-172-31-12-116/172.31.12.116
Start Time:       Mon, 30 Jun 2025 19:58:04 +0000
Labels:           app=tomcat-app-1
                  pod-template-hash=6b86c4b444
Annotations:      APPD_DEPLOYMENT_NAME: tomcat-app-1
                  APPD_INSTRUMENTED_CONTAINERS: tomcat-app-1
                  APPD_POD_INSTRUMENTATION_STATE: Successful
                  APPD_tomcat-app-1_APPNAME: tomcat-app-java-apps-1
                  APPD_tomcat-app-1_NODEID: 1588843
                  APPD_tomcat-app-1_NODENAME: tomcat-app-1--1
                  APPD_tomcat-app-1_TIERID: 31497
                  APPD_tomcat-app-1_TIERNAME: tomcat-app-1
                  cni.projectcalico.org/podIP: 10.244.116.150/32
                  cni.projectcalico.org/podIPs: 10.244.116.150/32
Status:           Running
IP:               10.244.116.150
IPs:
  IP:           10.244.116.150
Controlled By:  ReplicaSet/tomcat-app-1-6b86c4b444
Init Containers:
  appd-agent-attach-java:
    Container ID:  containerd://172d5f35af34468b82a4bd22a030495de959a6a71ae79bed17cae8d8d203adff
    Image:         docker.io/appdynamics/java-agent:latest
    Image ID:      docker.io/appdynamics/java-agent@sha256:d237aeb95a7b77d6e3e5b2c868e03cd22077e24424b58fa2380e6de340305e35
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
      cp -r /opt/appdynamics/. /opt/appdynamics-java && chown -R 999:999 /opt/appdynamics-java ; ls -la /opt/appdynamics-java
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 30 Jun 2025 19:58:05 +0000
      Finished:     Mon, 30 Jun 2025 19:58:10 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     20m
      memory:  75M
    Requests:
      cpu:        10m
      memory:     50M
    Environment:  <none>
    Mounts:
      /opt/appdynamics-java from appd-agent-repo-java (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-59w98 (ro)
Containers:
  tomcat-app-1:
    Container ID:   containerd://3234455f370b9af9c2e906ce0a0a183cf9812c53fd2fec3897a4a364c592031c
    Image:          docker.io/abhimanyubajaj98/tomcat-sample
    Image ID:       docker.io/abhimanyubajaj98/tomcat-sample@sha256:19558c877ec1fca506c3b7a6696a7a3a8914e855df3af93778323fb909b2fdff
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Mon, 30 Jun 2025 19:58:11 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      APPDYNAMICS_AGENT_ACCOUNT_ACCESS_KEY:           <set to the key 'controller-key' in secret 'cluster-agent-secret'>  Optional: false
      JAVA_TOOL_OPTIONS:                              -Xmx512m -Dappdynamics.agent.accountAccessKey=$(APPDYNAMICS_AGENT_ACCOUNT_ACCESS_KEY) -Dappdynamics.socket.collection.bci.enable=true -Dappdynamics.jvm.shutdown.mark.node.as.historical=false  -Dappdynamics.agent.reuse.nodeName=true -javaagent:/opt/appdynamics-java/javaagent.jar
      APPDYNAMICS_CONTROLLER_HOST_NAME:               ces-controller.saas.appdynamics.com
      APPDYNAMICS_AGENT_ACCOUNT_NAME:                 ces-controller
      APPDYNAMICS_AGENT_APPLICATION_NAME:             tomcat-app-java-apps-1
      APPDYNAMICS_JAVA_AGENT_REUSE_NODE_NAME_PREFIX:  tomcat-app-1
      APPDYNAMICS_CONTAINERINFO_FETCH_SERVICE:        cluster-metadata-service.appdynamics-1:9090
      APPDYNAMICS_CONTROLLER_PORT:                    443
      APPDYNAMICS_CONTROLLER_SSL_ENABLED:             true
      APPDYNAMICS_AGENT_TIER_NAME:                    tomcat-app-1
      APPDYNAMICS_POD_NAMESPACE:                      one-apps
      APPDYNAMICS_CONTAINER_NAME:                     tomcat-app-1
      APPDYNAMICS_NETVIZ_AGENT_HOST:                   (v1:status.hostIP)
      APPDYNAMICS_NETVIZ_AGENT_PORT:                  3892
    Mounts:
      /opt/appdynamics-java from appd-agent-repo-java (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-59w98 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  appd-agent-repo-java:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-59w98:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  32m   default-scheduler  Successfully assigned one-apps/tomcat-app-1-6b86c4b444-h65sj to ip-172-31-12-116
  Normal  Pulling    32m   kubelet            Pulling image "docker.io/appdynamics/java-agent:latest"
  Normal  Pulled     32m   kubelet            Successfully pulled image "docker.io/appdynamics/java-agent:latest" in 198ms (198ms including waiting). Image size: 84709497 bytes.
  Normal  Created    32m   kubelet            Created container: appd-agent-attach-java
  Normal  Started    32m   kubelet            Started container appd-agent-attach-java
  Normal  Pulling    32m   kubelet            Pulling image "docker.io/abhimanyubajaj98/tomcat-sample"
  Normal  Pulled     32m   kubelet            Successfully pulled image "docker.io/abhimanyubajaj98/tomcat-sample" in 236ms (236ms including waiting). Image size: 227949061 bytes.
  Normal  Created    32m   kubelet            Created container: tomcat-app-1
  Normal  Started    32m   kubelet            Started container tomcat-app-1

 

Viola!! This is what we wanted 🙂

Attachments
Version history
Last update:
‎06-30-2025 02:06 PM
Updated by:
Community Manager
Contributors