There will be times when multiple teams in your Organization will want to monitor and instrument their Applications. They would also want to have these applications report to different AppDynamics Backend (Controller) The latest Cluster agent releases gives you that option. Below has been tested on Cluster Agent 25.5 release.  First create 2 Applications: 1. tomcat-sample.yaml --- apiVersion: apps/v1 kind: Deployment metadata: name: tomcat-app labels: app: tomcat-app-java-apps namespace: java-apps spec: replicas: 1 selector: matchLabels: app: tomcat-app template: metadata: labels: app: tomcat-app spec: containers: - name: tomcat-app #image: docker.io/abhimanyubajaj98/tomcat-sample:latest image: docker.io/abhimanyubajaj98/tomcat-sample imagePullPolicy: Always ports: - containerPort: 8080 env: - name: JAVA_TOOL_OPTIONS value: "-Xmx512m" --- apiVersion: v1 kind: Service metadata: name: tomcat-app-service labels: app: tomcat-app namespace: java-apps spec: ports: - port: 8080 targetPort: 8080 selector: app: tomcat-app   2. tomcat-sample-ces.yaml --- apiVersion: apps/v1 kind: Deployment metadata: name: tomcat-app-1 labels: app: tomcat-app-java-apps-1 namespace: one-apps spec: replicas: 1 selector: matchLabels: app: tomcat-app-1 template: metadata: labels: app: tomcat-app-1 spec: containers: - name: tomcat-app-1 #image: docker.io/abhimanyubajaj98/tomcat-sample:latest image: docker.io/abhimanyubajaj98/tomcat-sample imagePullPolicy: Always ports: - containerPort: 8080 env: - name: JAVA_TOOL_OPTIONS value: "-Xmx512m" --- apiVersion: v1 kind: Service metadata: name: tomcat-app-service-1 labels: app: tomcat-app-1 namespace: one-apps spec: ports: - port: 8080 targetPort: 8080 selector: app: tomcat-app-1   To deploy-> 1. kubectl create ns java-apps one-apps 2. kubectl create -f tomcat-sample.yaml 3. kubectl create -f tomcat-sample-ces.yaml   Once deployed: root@ip-172-31-12-116:/opt/appdynamics/java-apps# kubectl -n java-apps get pods NAME READY STATUS RESTARTS AGE tomcat-app-76df69dc7b-f74wv 1/1 Running 0 71m root@ip-172-31-12-116:/opt/appdynamics/java-apps# kubectl -n one-apps get pods NAME READY STATUS RESTARTS AGE tomcat-app-1-6b86c4b444-h65sj 1/1 Running 0 20m   Auto-instrument For this I will deploy one cluster agent with Helm charts and the other one with normal kubectl command line. First up is creating the secret. Remember Cluster agent will be deployed in 2 namespaces so we will need to create 2 secrets in different namespace.  Our 1st namespace  : appdynamics Our 2nd namespace : appdynamics-1 Let's create secret: kubectl -n appdynamics create secret generic cluster-agent-secret --from-literal=controller-key=<access-key> kubectl -n appdynamics-1 create secret generic cluster-agent-secret --from-literal=controller-key=<access-key>   First Cluster Agent deployment in appdynamics namespace My helm: root@ip-172-31-12-116:/opt/appdynamics/cluster-agent/cluster-agent-alpine-arm64-bundled-distribution/helm-charts# cat values.yaml installClusterAgent: true installInfraViz: false installSplunkOtelCollector: false imageInfo: agentImage: docker.io/appdynamics/cluster-agent agentTag: 25.5.0-1126 operatorImage: docker.io/appdynamics/cluster-agent-operator operatorTag: 25.5.0-1107 imagePullPolicy: Always machineAgentImage: docker.io/appdynamics/machine-agent machineAgentTag: latest machineAgentWinImage: docker.io/appdynamics/machine-agent-analytics machineAgentWinTag: win-latest netVizImage: docker.io/appdynamics/machine-agent-netviz netvizTag: latest controllerInfo: url: https://controllerces.saas.appdynamics.com:443 account: controllerces username: null password: null accessKey: null globalAccount: controllerxxx_xxxxxx3 customSSLCert: null keyStorePasswordSecret: '' keyStoreFileSecret: '' authenticateProxy: false proxyUrl: null proxyUser: null proxyPassword: null createServiceAccount: true clusterAgent: containerProperties: containerBatchSize: 5 containerParallelRequestLimit: 1 containerRegistrationInterval: 120 logProperties: logFileSizeMb: 5 logFileBackups: 3 logLevel: DEBUG metricProperties: metricsSyncInterval: 30 metricUploadRetryCount: 2 metricUploadRetryIntervalMilliSeconds: 5 podMetricCollectionMaxGoRoutines: 3 podMetricCollectionRequestTimeoutSeconds: 5 nsToMonitorRegex: .* appName: two-ca instrumentationConfig: enabled: true containerAppCorrelationMethod: proxy instrumentationMethod: Env numberOfTaskWorkers: 5 appNameStrategy: label nsToInstrumentRegex: java-apps|nodejs-apps|dotnet-apps instrumentationRules: - namespaceRegex: java-apps language: java appNameLabel: app runAsUser: 999 runAsGroup: 999 imageInfo: image: docker.io/appdynamics/java-agent:latest agentMountPath: /opt/appdynamics imagePullPolicy: Always   helm install -f values.yaml cluster-agent -n appdynamcis  root@ip-172-31-12-116:/opt/appdynamics/cluster-agent/cluster-agent-alpine-arm64-bundled-distribution/helm-charts# kubectl -n appdynamics get pods NAME READY STATUS RESTARTS AGE appdynamics-operator-769dcf8f4b-kjm2z 1/1 Running 0 77m two-ca-appdynamics-54bc66f55c-5qzfk 1/1 Running 0 77m   Second Cluster Agent deployment in appdynamics namespace Before we deploy the second Cluster Agent, we will need to edit our cluster-agent-operator.yaml file. What we need to make sure is every place where you have  namespace: appdynamics This gets changed to namespace: appdynamics-1   I have uploaded cluster-agent-operator.txt file that has this configuration. You can use with 25.5 version of Cluster Agent by changing the extension to yaml Do-> kubectl create -f cluster-agent-operator.yaml My cluster-agent.yaml: root@ip-172-31-12-116:/opt/appdynamics/cluster-agent/cluster-agent-alpine-arm64-bundled-distribution# cat cluster-agent.yaml apiVersion: cluster.appdynamics.com/v1alpha1 kind: Clusteragent metadata: name: k8s-cluster-agent namespace: appdynamics-1 spec: appName: "cluster-2" controllerUrl: "https://ces-controller.saas.appdynamics.com:443" account: "ces-controller" # docker image info image: "docker.io/appdynamics/cluster-agent:latest" serviceAccountName: appdynamics-cluster-agent nsToMonitorRegex: appdynamics resources: limits: cpu: 90m memory: "200Mi" requests: cpu: 90m memory: "100Mi" instrumentationMethod: Env nsToInstrumentRegex: one-apps appNameStrategy: label instrumentationRules: - namespaceRegex: one-apps language: java appNameLabel: app runAsUser: 999 runAsGroup: 999 imageInfo: image: docker.io/appdynamics/java-agent:latest agentMountPath: /opt/appdynamics imagePullPolicy: Always ### Uncomment the following line if you need pull secret #imagePullSecret: "<your-docker-pull-secret-name>" ### Uncomment the following line if you need to enable the profiling #pprofEnabled: true #pprofPort: 9991   kubectl create -f cluster-agent.yaml Once done: root@ip-172-31-12-116:/opt/appdynamics/cluster-agent/cluster-agent-alpine-arm64-bundled-distribution# kubectl -n appdynamics-1 get pods NAME READY STATUS RESTARTS AGE appdynamics-operator-7677764d7c-52nx9 1/1 Running 0 51m k8s-cluster-agent-6dbdb6b4dc-dkdgs 1/1 Running 0 30m   Now, what should happen is namespace one-apps should be instrumented with second Cluster Agent aka one running in appdynamics-1 namespace and java-apps should be instrumented with first Cluster Agent aka one running in appdynamics namespace And that is what happened-> root@ip-172-31-12-116:/opt/appdynamics/cluster-agent/cluster-agent-alpine-arm64-bundled-distribution# kubectl -n java-apps describe pod Name: tomcat-app-76df69dc7b-f74wv Namespace: java-apps Priority: 0 Service Account: default Node: ip-172-31-12-116/172.31.12.116 Start Time: Mon, 30 Jun 2025 19:07:11 +0000 Labels: app=tomcat-app pod-template-hash=76df69dc7b Annotations: APPD_DEPLOYMENT_NAME: tomcat-app APPD_INSTRUMENTED_CONTAINERS: tomcat-app APPD_POD_INSTRUMENTATION_STATE: Successful APPD_tomcat-app_APPNAME: tomcat-app-java-apps APPD_tomcat-app_NODEID: 1588804 APPD_tomcat-app_NODENAME: tomcat-app--12 APPD_tomcat-app_TIERID: 30171 APPD_tomcat-app_TIERNAME: tomcat-app cni.projectcalico.org/podIP: 10.244.116.136/32 cni.projectcalico.org/podIPs: 10.244.116.136/32 Status: Running IP: 10.244.116.136 IPs: IP: 10.244.116.136 Controlled By: ReplicaSet/tomcat-app-76df69dc7b Init Containers: appd-agent-attach-java: Container ID: containerd://2785c3422edd754d96270af5e970312791079bb6dc9840acd3ab4af84e5985a0 Image: docker.io/appdynamics/java-agent:latest Image ID: docker.io/appdynamics/java-agent@sha256:d237aeb95a7b77d6e3e5b2c868e03cd22077e24424b58fa2380e6de340305e35 Port: <none> Host Port: <none> Command: /bin/sh -c cp -r /opt/appdynamics/. /opt/appdynamics-java && chown -R 999:999 /opt/appdynamics-java ; ls -la /opt/appdynamics-java State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 30 Jun 2025 19:07:15 +0000 Finished: Mon, 30 Jun 2025 19:07:20 +0000 Ready: True Restart Count: 0 Limits: cpu: 20m memory: 75M Requests: cpu: 10m memory: 50M Environment: <none> Mounts: /opt/appdynamics-java from appd-agent-repo-java (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-95g58 (ro) Containers: tomcat-app: Container ID: containerd://5a27c1bb0d2f539f8288fc65a99cebabeed69d10d34de85c50d0f1389c4e5701 Image: docker.io/abhimanyubajaj98/tomcat-sample Image ID: docker.io/abhimanyubajaj98/tomcat-sample@sha256:19558c877ec1fca506c3b7a6696a7a3a8914e855df3af93778323fb909b2fdff Port: 8080/TCP Host Port: 0/TCP State: Running Started: Mon, 30 Jun 2025 19:07:20 +0000 Ready: True Restart Count: 0 Environment: APPDYNAMICS_AGENT_ACCOUNT_ACCESS_KEY: <set to the key 'controller-key' in secret 'cluster-agent-secret'> Optional: false JAVA_TOOL_OPTIONS: -Xmx512m -Dappdynamics.agent.accountAccessKey=$(APPDYNAMICS_AGENT_ACCOUNT_ACCESS_KEY) -Dappdynamics.socket.collection.bci.enable=true -Dappdynamics.jvm.shutdown.mark.node.as.historical=false -Dappdynamics.agent.reuse.nodeName=true -javaagent:/opt/appdynamics-java/javaagent.jar APPDYNAMICS_CONTROLLER_HOST_NAME: controllerces.saas.appdynamics.com APPDYNAMICS_CONTROLLER_PORT: 443 APPDYNAMICS_AGENT_TIER_NAME: tomcat-app APPDYNAMICS_POD_NAMESPACE: java-apps APPDYNAMICS_CONTAINER_NAME: tomcat-app APPDYNAMICS_CONTROLLER_SSL_ENABLED: true APPDYNAMICS_AGENT_ACCOUNT_NAME: controllerces APPDYNAMICS_AGENT_APPLICATION_NAME: tomcat-app-java-apps APPDYNAMICS_JAVA_AGENT_REUSE_NODE_NAME_PREFIX: tomcat-app APPDYNAMICS_CONTAINERINFO_FETCH_SERVICE: cluster-metadata-service.appdynamics:9090 APPDYNAMICS_NETVIZ_AGENT_HOST: (v1:status.hostIP) APPDYNAMICS_NETVIZ_AGENT_PORT: 3892 Mounts: /opt/appdynamics-java from appd-agent-repo-java (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-95g58 (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: appd-agent-repo-java: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> kube-api-access-95g58: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: <none>   Name: tomcat-app-1-6b86c4b444-h65sj Namespace: one-apps Priority: 0 Service Account: default Node: ip-172-31-12-116/172.31.12.116 Start Time: Mon, 30 Jun 2025 19:58:04 +0000 Labels: app=tomcat-app-1 pod-template-hash=6b86c4b444 Annotations: APPD_DEPLOYMENT_NAME: tomcat-app-1 APPD_INSTRUMENTED_CONTAINERS: tomcat-app-1 APPD_POD_INSTRUMENTATION_STATE: Successful APPD_tomcat-app-1_APPNAME: tomcat-app-java-apps-1 APPD_tomcat-app-1_NODEID: 1588843 APPD_tomcat-app-1_NODENAME: tomcat-app-1--1 APPD_tomcat-app-1_TIERID: 31497 APPD_tomcat-app-1_TIERNAME: tomcat-app-1 cni.projectcalico.org/podIP: 10.244.116.150/32 cni.projectcalico.org/podIPs: 10.244.116.150/32 Status: Running IP: 10.244.116.150 IPs: IP: 10.244.116.150 Controlled By: ReplicaSet/tomcat-app-1-6b86c4b444 Init Containers: appd-agent-attach-java: Container ID: containerd://172d5f35af34468b82a4bd22a030495de959a6a71ae79bed17cae8d8d203adff Image: docker.io/appdynamics/java-agent:latest Image ID: docker.io/appdynamics/java-agent@sha256:d237aeb95a7b77d6e3e5b2c868e03cd22077e24424b58fa2380e6de340305e35 Port: <none> Host Port: <none> Command: /bin/sh -c cp -r /opt/appdynamics/. /opt/appdynamics-java && chown -R 999:999 /opt/appdynamics-java ; ls -la /opt/appdynamics-java State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 30 Jun 2025 19:58:05 +0000 Finished: Mon, 30 Jun 2025 19:58:10 +0000 Ready: True Restart Count: 0 Limits: cpu: 20m memory: 75M Requests: cpu: 10m memory: 50M Environment: <none> Mounts: /opt/appdynamics-java from appd-agent-repo-java (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-59w98 (ro) Containers: tomcat-app-1: Container ID: containerd://3234455f370b9af9c2e906ce0a0a183cf9812c53fd2fec3897a4a364c592031c Image: docker.io/abhimanyubajaj98/tomcat-sample Image ID: docker.io/abhimanyubajaj98/tomcat-sample@sha256:19558c877ec1fca506c3b7a6696a7a3a8914e855df3af93778323fb909b2fdff Port: 8080/TCP Host Port: 0/TCP State: Running Started: Mon, 30 Jun 2025 19:58:11 +0000 Ready: True Restart Count: 0 Environment: APPDYNAMICS_AGENT_ACCOUNT_ACCESS_KEY: <set to the key 'controller-key' in secret 'cluster-agent-secret'> Optional: false JAVA_TOOL_OPTIONS: -Xmx512m -Dappdynamics.agent.accountAccessKey=$(APPDYNAMICS_AGENT_ACCOUNT_ACCESS_KEY) -Dappdynamics.socket.collection.bci.enable=true -Dappdynamics.jvm.shutdown.mark.node.as.historical=false -Dappdynamics.agent.reuse.nodeName=true -javaagent:/opt/appdynamics-java/javaagent.jar APPDYNAMICS_CONTROLLER_HOST_NAME: ces-controller.saas.appdynamics.com APPDYNAMICS_AGENT_ACCOUNT_NAME: ces-controller APPDYNAMICS_AGENT_APPLICATION_NAME: tomcat-app-java-apps-1 APPDYNAMICS_JAVA_AGENT_REUSE_NODE_NAME_PREFIX: tomcat-app-1 APPDYNAMICS_CONTAINERINFO_FETCH_SERVICE: cluster-metadata-service.appdynamics-1:9090 APPDYNAMICS_CONTROLLER_PORT: 443 APPDYNAMICS_CONTROLLER_SSL_ENABLED: true APPDYNAMICS_AGENT_TIER_NAME: tomcat-app-1 APPDYNAMICS_POD_NAMESPACE: one-apps APPDYNAMICS_CONTAINER_NAME: tomcat-app-1 APPDYNAMICS_NETVIZ_AGENT_HOST: (v1:status.hostIP) APPDYNAMICS_NETVIZ_AGENT_PORT: 3892 Mounts: /opt/appdynamics-java from appd-agent-repo-java (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-59w98 (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: appd-agent-repo-java: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> kube-api-access-59w98: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 32m default-scheduler Successfully assigned one-apps/tomcat-app-1-6b86c4b444-h65sj to ip-172-31-12-116 Normal Pulling 32m kubelet Pulling image "docker.io/appdynamics/java-agent:latest" Normal Pulled 32m kubelet Successfully pulled image "docker.io/appdynamics/java-agent:latest" in 198ms (198ms including waiting). Image size: 84709497 bytes. Normal Created 32m kubelet Created container: appd-agent-attach-java Normal Started 32m kubelet Started container appd-agent-attach-java Normal Pulling 32m kubelet Pulling image "docker.io/abhimanyubajaj98/tomcat-sample" Normal Pulled 32m kubelet Successfully pulled image "docker.io/abhimanyubajaj98/tomcat-sample" in 236ms (236ms including waiting). Image size: 227949061 bytes. Normal Created 32m kubelet Created container: tomcat-app-1 Normal Started 32m kubelet Started container tomcat-app-1   Viola!! This is what we wanted 🙂 ... View more

The AppDynamics Smart Agent offers a powerful way to automate agent installation, updates, and management. While it supports default artifact repositories, you can also host your own agent binaries and use the Custom HTTP URL option to install agents — great for enterprise customization and version control. In this article, we’ll walk through how to host agents in Amazon S3 and use the Smart Agent to install them via a custom HTTP URL.   Prerequisites Before we start, ensure: You have access to an Amazon S3 bucket The Smart Agent is installed on your host The agent ZIP file (e.g., Machine Agent, Java Agent) is available locally   Step 1: Upload the Agent to S3 Use the AWS CLI to upload your agent ZIP file to S3: aws s3 cp machineagent-bundle-64bit-linux-25.4.0.4712.zip s3://your-bucket-name/ Replace your-bucket-name with your actual S3 bucket.   Step 2: Make the Object Publicly Accessible To allow Smart Agent (which runs in AppDynamics cloud) to download the file, it must be public. Allow Public Access via ACL: aws s3api put-object-acl \ --bucket your-bucket-name \ --key machineagent-bundle-64bit-linux-25.4.0.4712.zip \ --acl public-read   Step 3: Get the Public HTTP URL Public S3 objects are accessible at: https://<bucket-name>.s3.<region>.amazonaws.com/<object-key> Example: https://appd-agent-smart.s3.us-west-2.amazonaws.com/machineagent-bundle-64bit-linux-25.4.0.4712.zip   Step 4: Use the URL in Smart Agent Go to AppDynamics Controller → Smart Agent UI Select Install Agent Choose the agent type (e.g., Machine Agent) Under Custom HTTP URL, paste the S3 link       Result Smart Agent will now download the ZIP from your public S3 URL and install the agent on the target host.   ... View more

Overview When monitoring Kubernetes workloads using AppDynamics' Cluster Agent, users often observe a mismatch or confusion between the Memory (MB) values in the table and the Usage% shown in pod detail graphs. This article explains how CPU and memory metrics are computed, displayed, and what to expect when resource limits are or are not defined.   Ask 1: How is CPU% Busy calculated for Pods? AppDynamics uses the following logic: If CPU limits are defined in the pod spec, the CPU%Busy is calculated relative to the defined limit. If limits are not set, AppDynamics falls back to node-level CPU usage, and percentage values may not reflect actual pod utilization.   Ask 2: Does the same logic apply to Memory (MB)? Not exactly. Memory (MB) is always collected as the raw value used by the container. Memory usage % (Usage%) is only calculated and displayed if a memory limit is defined. If no memory limit is set, AppDynamics may still show MB used in the table view, but the Usage% will remain at 0%, as no denominator is available for calculation.   Ask 3: Why does the graph show 0% even when memory is consumed? This behavior is expected when no resource limits are set. AppDynamics cannot calculate a percentage of memory used if there's no defined upper limit to compare against.   Screenshot Summary: Screenshot 1: Pod list view Shows the AppDynamics Operator pod using 11.84 MB memory with 1% CPU.    2. Screenshot 2: Pod container view (with limits defined) Graph shows 11.91% memory usage out of 100Mi defined.          3. Screenshot 3: Pod container view (without limits) NodeJS pod consuming 33 MB memory but graph shows 0% usage.       Verifying Pod Resource Limits: To check if a pod has limits defined: kubectl get pod <pod-name> -n <namespace> -o jsonpath="{.spec.containers[*].resources.limits}"   If output is empty ({}), no limits are set, and usage% will show 0%.   Conclusion: Always refer to the table for actual memory used (MB). Usage% depends on resource limits being set. No bug here – this is by design, and understanding this helps interpret the UI accurately.   Recommendations: Set CPU and memory limits in pod specs for better visibility and scaling insights. Use kubectl top pod to cross-verify values if needed. ... View more

AWS Lambda auto-scales, but every account & region has a concurrency ceiling. When you slam into it CloudWatch emits the Throttles metric. Surfacing that (plus Invocations, Errors, Duration…) in AppDynamics lets you keep all observability in one place and build health rules so you can react before customers feel pain. Pre-requisite AppDynamics Controller Machine Agent EC2 instance profile (IAM role with cloudwatch:ListMetrics and cloudwatch:GetMetricStatistics only) or AWS AccessKey and Secret Java 8 + Maven for building the extension High‑level flow   Install the Machine Agent (recap) wget https://download.appdynamics.com/.../MachineAgent-25.x.x.zip unzip MachineAgent-25.x.x.zip -d /opt/appdynamics # 2 Point to your Controller vim /opt/appdynamics/conf/controller-info.xml # <controller-host>my-controller.company.com</controller-host> # <controller-port>443</controller-port> # <account-name>customer1</account-name> # <account-access-key>abcd123…</account-access-key> # 3 (optional) register as a service sudo systemctl enable appdynamics-machine-agent sudo systemctl start appdynamics-machine-agent Steps are noted here: https://docs.appdynamics.com/appd/24.x/25.4/en/infrastructure-visibility/machine-agent/install-the-machine-agent Build the Custom Namespace extension Extension we are using: https://developer.cisco.com/codeexchange/github/repo/Appdynamics/aws-customnamespace-monitoring-extension/ git clone https://github.com/Appdynamics/aws-customnamespace-monitoring-extension.git cd aws-customnamespace-monitoring-extension mvn clean package cp target/AWSCustomNamespaceMonitor-*.zip \ /opt/appdynamics/monitors/ cd /opt/appdynamics/monitors unzip AWSCustomNamespaceMonitor-*.zip This creates AWSCustomNamespaceMonitor/ with configuration files you need Authoring config.yml metricPrefix: "Custom Metrics|AWS Lambda|" namespace: "AWS/Lambda" accounts: - displayAccountName: "prod" regions: ["us-east-2"] # your Lambda region # keys can stay empty on an EC2 instance profile # awsAccessKey: # awsSecretKey: # optional – pulls every function; give explicit names to narrow it dimensions: - name: "FunctionName" displayName: "Function" values: [".*"] # Display all Lambda functions regionEndPoints: #us-east-1 : monitoring.us-east-1.amazonaws.com us-east-2 : monitoring.us-east-2.amazonaws.com cloudWatchMonitoring: "Basic" concurrencyConfig: noOfAccountThreads: 3 noOfRegionThreadsPerAccount: 3 noOfMetricThreadsPerRegion: 3 threadTimeOut: 30 # Global metrics config for all accounts metricsConfig: # By default, all metrics retrieved from cloudwatch are 'Average' values. # This option allows you to override the metric type. # metricName supports regex # Allowed statTypes are: ave, max, min, sum, samplecount # # Note: Irrespective of the metric type, value will still be reported as # Observed value to the Controller includeMetrics: - name: "Invocations" statType: "sum" delta: false multiplier: 1 aggregationType: "OBSERVATION" timeRollUpType: "AVERAGE" clusterRollUpType: "INDIVIDUAL" - name: "Errors" # alias: "Errors" statType: "sum" delta: false multiplier: 1 aggregationType: "OBSERVATION" timeRollUpType: "AVERAGE" clusterRollUpType: "INDIVIDUAL" - name: "Duration" # alias: "Duration" statType: "ave" delta: false multiplier: 1 aggregationType: "OBSERVATION" timeRollUpType: "AVERAGE" clusterRollUpType: "INDIVIDUAL" - name: "Throttles" # alias: "Throttles" statType: "sum" delta: false multiplier: 1 aggregationType: "OBSERVATION" timeRollUpType: "AVERAGE" clusterRollUpType: "INDIVIDUAL" - name: "DeadLetterErrors" # alias: "DeadLetterErrors" statType: "ave" delta: false multiplier: 1 aggregationType: "OBSERVATION" timeRollUpType: "AVERAGE" clusterRollUpType: "INDIVIDUAL" - name: "Availability" # alias: "Availability" statType: "ave" delta: false multiplier: 1 aggregationType: "OBSERVATION" timeRollUpType: "AVERAGE" clusterRollUpType: "INDIVIDUAL" - name: "IteratorAge" # alias: "IteratorAge" statType: "max" delta: false multiplier: 1 aggregationType: "OBSERVATION" timeRollUpType: "AVERAGE" clusterRollUpType: "INDIVIDUAL" metricsTimeRange: startTimeInMinsBeforeNow: 5 endTimeInMinsBeforeNow: 0 # Rate limit ( per second ) for GetMetricStatistics, default value is 400. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_limits.html getMetricStatisticsRateLimit: 400 maxErrorRetrySize: 0 All the metrics that you can configure are listed here: Types of metrics for Lambda functions Learn about the different types of metrics for Lambda: invocation metrics, performance metrics, concurrency metrics…docs.aws.amazon.com Restart & verify sudo systemctl restart appdynamics-machine-agent tail -f /opt/appdynamics/logs/machine-agent.log | grep -i "AWS/Lambda" DEBUG AMonitorJob-AWSCustomNamespaceMonitor - Printing Metric [OBSERVATION/AVERAGE/INDIVIDUAL] [Custom Metrics|AWS Lambda|AWS/Lambda|prod|us-east-2|Function|appdshield-deploy-agent|Invocations]=[1]]   With ~10 minutes you can bring Lambda’s key CloudWatch metrics into AppDynamics, build proactive health rules, and keep your SRE team on a single pane of glass. Extend the same approach to other AWS namespaces or use AppDynamics’ Lambda‑specific extension if you want the full metric set out‑of‑the‑box. Happy monitoring! ... View more

How to Deploy AppDynamics Database Agent for MySQL on OpenShift Introduction: Monitoring database performance is crucial for ensuring application stability and efficiency. This guide walks you through deploying the AppDynamics Database Agent on OpenShift to monitor a MySQL database. You'll learn how to: Set up a MySQL instance on OpenShift Deploy the AppDynamics Database Agent with proper permissions Configure MySQL monitoring in the AppDynamics Controller UI Verify metrics and troubleshoot potential issues By the end of this guide, you’ll have full visibility into your MySQL performance, including query analysis, resource utilization, and overall health monitoring in AppDynamics.   1. Prerequisites OpenShift Cluster: Access via the oc CLI or the OpenShift Web Console. AppDynamics Controller: You have the host, port, account name, access key. AppDynamics License: Ensure Database Monitoring (MySQL) is licensed. Database Agent Image: Typically docker.io/appdynamics/db-agent or a registry provided by AppDynamics. Basic MySQL Docker image: For this example, we’ll use mysql:8.0 from Docker Hub. 2. Create/Open a New Project In the CLI, for example: oc new-project mydb-project oc project mydb-project 3. Deploy a MySQL Instance on OpenShift Below is a simple YAML for a Deployment and Service. In a real environment, you’d likely set up persistent storage, handle secrets for credentials, etc. # mysql.yaml apiVersion: apps/v1 kind: Deployment metadata: name: mysql-deployment spec: replicas: 1 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - name: mysql image: mysql:8.0 env: - name: MYSQL_ROOT_PASSWORD value: rootpassword - name: MYSQL_DATABASE value: sampledb - name: MYSQL_USER value: appduser - name: MYSQL_PASSWORD value: appdpass ports: - containerPort: 3306 --- apiVersion: v1 kind: Service metadata: name: mysql-service spec: selector: app: mysql ports: - protocol: TCP port: 3306 targetPort: 3306 Apply it: oc apply -f mysql.yaml Check that the pod and service are running: oc get pods oc get svc You should see mysql-deployment-... running and a mysql-service on port 3306 4. Create a Secret for AppDynamics Credentials Store your AppDynamics account access key securely. For example: oc create secret generic appd-secret \ --from-literal=APPDYNAMICS_AGENT_ACCOUNT_ACCESS_KEY=<YOUR-ACCESS-KEY> This secret will be referenced by the Database Agent deployment. 5. Grant the “anyuid” SCC The AppDynamics Database Agent container often tries to run a startup script (start-appdynamics-database-monitoring). In OpenShift’s default restricted SCC, containers run as a random non-root UID. If the script in the image doesn’t have proper permissions for non-root, you get a permission denied error. 5.1 Create a Service Account (Recommended) oc create serviceaccount appd-db-agent-sa 5.2 Give anyuid SCC to That Service Account oc adm policy add-scc-to-user anyuid -z appd-db-agent-sa -n mydb-project Now, the service account appd-db-agent-sa is allowed to run images that require root or any specific UID. Note: This is a quick fix and less secure. A better approach is to build a custom image or ensure the official image has proper file permissions (so it can run under restricted SCC). But for simplicity, we’ll use anyuid in this guide. 6. Deploy the AppDynamics Database Agent Here’s a minimal YAML that uses environment variables (rather than a config file) to configure the agent. It references the secret for the access key and uses the appd-db-agent-sa service account we just created. # appd-db-agent.yaml apiVersion: apps/v1 kind: Deployment metadata: name: appd-database-agent spec: replicas: 1 selector: matchLabels: app: appd-database-agent template: metadata: labels: app: appd-database-agent spec: serviceAccountName: appd-db-agent-sa # <-- Important! containers: - name: db-agent image: docker.io/appdynamics/db-agent:latest env: - name: APPDYNAMICS_CONTROLLER_HOST_NAME value: "my-controller.example.com" - name: APPDYNAMICS_CONTROLLER_PORT value: "443" - name: APPDYNAMICS_CONTROLLER_SSL_ENABLED value: "true" - name: APPDYNAMICS_AGENT_ACCOUNT_NAME value: "myAccount" - name: APPDYNAMICS_AGENT_ACCOUNT_ACCESS_KEY valueFrom: secretKeyRef: name: appd-secret key: APPDYNAMICS_AGENT_ACCOUNT_ACCESS_KEY # Name this agent as it appears in the Controller - name: APPDYNAMICS_DB_AGENT_NAME value: "MyOpenShiftDBAgent" - name: APPDYNAMICS_DB_AGENT_TIER_NAME value: "MyDBTier" - name: APPDYNAMICS_DB_AGENT_NODE_NAME value: "MyDBNode" # For memory management or other Java opts - name: JAVA_OPTS value: "-Xms128m -Xmx512m" Apply it: oc apply -f appd-db-agent.yaml Check pod logs: oc get pods oc logs deployment/appd-database-agent If all goes well, you should see logs indicating it connected to the Controller. If you get a permission denied error, confirm you’re using the serviceAccountName: appd-db-agent-sa and that the SA has the anyuid SCC. 7. Configure MySQL Monitoring in the Controller UI By default, if you only used environment variables, the agent knows how to contact the Controller but doesn’t yet know which MySQL host to monitor. We create a MySQL collector in the AppDynamics UI: Open the AppDynamics Controller in your browser. Navigate to Databases → Agents. Find your agent (e.g., MyOpenShiftDBAgent) and click on it. In the Databases tab, click Add → Add MySQL Collector. Collector Configuration: Display Name: e.g. MyOpenShiftDBAgent Host: mysql-service (assuming your Database Agent is in the same namespace as mysql-service), If in a different namespace, you might need the FQDN: mysql-service.mydb-project.svc.cluster.local Port: 3306 Username: appduser Password: appdpass (Optionally) Tweak advanced fields like Collection Interval, Lock Level, Query Analysis, etc.   Save the collector. Within a couple of minutes, the Database Agent should start collecting MySQL metrics. 8. Verify and View Metrics Once the MySQL collector is set up: In the AppDynamics Controller, go to Databases → Databases. You’ll see a row for your new MySQL collector (e.g., MyOpenShiftDBAgent). Click it to see: Overview: CPU, memory, throughput. Queries: Slow queries, wait stats, top queries, etc. Resources: For deeper performance insights.   Conclusion Deploying the AppDynamics Database Agent alongside MySQL on OpenShift involves: Creating a project, a MySQL deployment, and a service. Setting up the Database Agent container, ensuring it can run the startup script (via SCC or a custom image). Creating a MySQL collector in AppDynamics to specify connection details. Once configured, you’ll see full visibility into your MySQL instance — top queries, performance metrics, and more — in the AppDynamics Controller. Feel free to add screenshots to illustrate each step in both the OpenShift Web Console and the AppDynamics UI. ... View more

Guide to Monitoring URLs with Authentication Using Splunk AppDynamics and Python Monitoring URLs are an important part of your FullStackMonitoring. Splunk AppDynamics lets you monitor URLs with different authentication. In this article, we will create a simple URL with a username and password. Afterwards, we will monitor it using AppDynamics Machine Agent. Create a Simple API with Python (Flask) Install Flask: pip install flask​ Create the API: Save the following Python code to a file, e.g., basic_auth_api.py : from flask import Flask, request, jsonify from flask_httpauth import HTTPBasicAuth app = Flask(__name__) auth = HTTPBasicAuth() # Dummy users for authentication users = { "user1": "password123", "user2": "securepassword", } @auth.get_password def get_pw(username): return users.get(username) @app.route('/api/data', methods=['GET']) @auth.login_required def get_data(): return jsonify({"message": f"Hello, {auth.username()}! Here is your data."}) if __name__ == '__main__': app.run(debug=True, port=5000) Run the API: Start the server by running: python basic_auth_api.py Test the API: Use curl to access the API: curl -u user1:password123 http://127.0.0.1:5000/api/data You should see a response like this: { "message": "Hello, user1! Here is your data." }​ Install Machine Agent You can install the Machine agent as recommended here Setup URL Monitoring Extension Clone the Github Repo: git clone https://github.com/Appdynamics/url-monitoring-extension.git​ cd url-monitoring-extension​ Download and install Apache Maven which is configured with  Java 8  to build the extension artifact from the source. You can check the Java version used in Maven using command  mvn -v  or  mvn --version . If your maven is using some other Java version then please download Java 8 for your platform and set JAVA_HOME parameter before starting maven. Run below in url-monitoring-extension directory mvn clean install Go into the target directory and copy the UrlMonitor-2.2.1.zip, Afterwards unzip the content inside <MA-Home>/monitors/folder cd target/ mv UrlMonitor-2.2.1.zip /opt/appdynamics/machine-agent/monitors unzip UrlMonitor-2.2.1.zip​ This will create an UrlMonitor directory inside the Monitors folder. Monitor the URL Inside the UrlMonitor folder, edit the config.yml file Under sites, I have added: sites: - name: AppDynamics url: http://127.0.0.1:5000/api/data username: user1 password: password123 authType: BASIC​ Change: metricPrefix: "Custom Metrics|URL Monitor|"​ Now, All you need to do is Start your Machine Agent again. Afterward, you can see this URL monitor in your AppDynamics Controller. ... View more

A Step-by-Step Guide to Setting Up and Monitoring Redis with AppDynamics on Ubuntu EC2 Monitoring your Redis instance is essential for ensuring optimal performance and identifying potential bottlenecks in real-time. In this guide, we’ll walk through the process of setting up Redis on an Ubuntu EC2 instance and configuring SplunkAppDynamics Redis Monitoring Extension to capture key metrics. Step 1: Setting up Redis on Ubuntu Prerequisites An AWS account with an EC2 instance running Ubuntu. SSH access to your EC2 instance. Installing Redis Update package lists and install Redis: sudo apt-get update sudo apt-get install redis-server​ Verify the installation: redis-server --version​ Ensure Redis is running: sudo systemctl status redis​ Step 2: Installing AppDynamics Machine Agent Download the Machine Agent: Visit AppDynamics and download the Machine Agent for your environment. Install the Machine Agent: Follow the installation steps provided in the AppDynamics Machine Agent documentation. https://docs.appdynamics.com/appd/24.x/24.11/en/infrastructure-visibility/machine-agent/install-the-machine-agent Verify Installation: Start the Machine Agent and confirm it connects to your AppDynamics Controller. Step 3: Configuring AppDynamics Redis Monitoring Extension Clone the Redis Monitoring Extension Repository git clone https://github.com/Appdynamics/redis-monitoring-extension.git cd redis-monitoring-extension Build the Extension sudo apt-get install openjdk-8-jdk maven mvn clean install Locate the  .zip file in the target folder and extract it: unzip target/RedisMonitor-*.zip -d <MachineAgent_Dir>/monitors/ Edit the Configuration File Navigate to the extracted folder and edit config.yml : metricPrefix: "Custom Metrics|Redis" #Add your list of Redis servers here. servers: - name: "localhost" host: "localhost" port: "6379" password: "" #encryptedPassword: "" useSSL: false Restart the Machine Agent .<MachineAgent_Dir>/bin/machine-agent Step 4: Verifying Metrics in AppDynamics Log in to your AppDynamics Controller. Navigate to the Metric Browser. Look for metrics under the path: Custom Metrics|Redis Verify that metrics like used_memory , connected_clients , and keyspace_hits are visible. Conclusion By combining the power of Redis with the advanced monitoring capabilities of AppDynamics, you can ensure your application remains scalable and responsive under varying workloads. Whether you’re troubleshooting an issue or optimizing performance, this setup gives you full visibility into your Redis instance. If you found this guide helpful, please share and connect with me for more DevOps insights! ... View more

Comprehensive Guide to RabbitMQ Setup, Integration with Python, and Monitoring with AppDynamics Introduction RabbitMQ is a powerful open-source message broker that supports a variety of messaging protocols, including AMQP. It allows developers to build robust, scalable, and asynchronous messaging systems. However, to ensure optimal performance, monitoring RabbitMQ metrics is crucial. This tutorial walks you through setting up RabbitMQ, integrating it with a Python application, and monitoring its metrics using AppDynamics. Step 1: Setting Up RabbitMQ 1.1 Install RabbitMQ via Docker To quickly get RabbitMQ up and running, use the official RabbitMQ Docker image with the management plugin enabled. Run the following command to start RabbitMQ: docker run -d --hostname my-rabbit --name rabbitmq \ -e RABBITMQ_DEFAULT_USER=guest \ -e RABBITMQ_DEFAULT_PASS=guest \ -p 5672:5672 -p 15672:15672 \ rabbitmq:management Management Console: Accessible at http://localhost:15672 . Default Credentials: Username: guest Password: guest 1.2 Verify the Setup Once the container is running, verify the RabbitMQ server by accessing the Management Console in your browser. Alternatively, test the API endpoint: curl -u guest:guest http://localhost:15672/api/overview This should return RabbitMQ metrics in JSON format. Step 2: Writing a Simple RabbitMQ Producer and Consumer in Python 2.1 Install Required Library Install the pika library for Python, which is used to interact with RabbitMQ: pip install pika 2.2 Create the Producer Script ( send.py ) This script connects to RabbitMQ, declares a queue, and sends a message. import pika # Connect to RabbitMQ connection = pika.BlockingConnection(pika.ConnectionParameters('localhost')) channel = connection.channel() # Declare a queue channel.queue_declare(queue='hello') # Publish a message channel.basic_publish(exchange='', routing_key='hello', body='Hello RabbitMQ!') print(" [x] Sent 'Hello RabbitMQ!'") connection.close() 2.3 Create the Consumer Script ( receive.py ) This script connects to RabbitMQ, consumes messages from the queue, and prints them. import pika # Connect to RabbitMQ connection = pika.BlockingConnection(pika.ConnectionParameters('localhost')) channel = connection.channel() # Declare a queue channel.queue_declare(queue='hello') # Define a callback to process messages def callback(ch, method, properties, body): print(f" [x] Received {body}") channel.basic_consume(queue='hello', on_message_callback=callback, auto_ack=True) print(' [*] Waiting for messages. To exit press CTRL+C') channel.start_consuming() 2.4 Test the Application a. Run the consumer in one terminal: python3 receive.py b. Send a message from another terminal: python3 send.py c. Observe the message output in the consumer terminal. [x] Sent 'Hello RabbitMQ!' [x] Received b'Hello RabbitMQ!' Step 3: Monitoring RabbitMQ with AppDynamics 3.1 Configure RabbitMQ Management Plugin Ensure that the RabbitMQ Management Plugin is enabled (default in the Docker image). It exposes an HTTP API that provides metrics. 3.2 Create a Custom Monitoring Script Use a shell script to fetch RabbitMQ metrics and send them to the AppDynamics Machine Agent. script.sh #!/bin/bash # RabbitMQ Management API credentials USERNAME="guest" PASSWORD="guest" URL="http://localhost:15672/api/overview" # Fetch metrics from RabbitMQ Management API RESPONSE=$(curl -s -u $USERNAME:$PASSWORD $URL) if [[ $? -ne 0 || -z "$RESPONSE" ]]; then echo "Error: Unable to fetch RabbitMQ metrics" exit 1 fi MESSAGES=$(echo "$RESPONSE" | jq '.queue_totals.messages // 0') MESSAGES_READY=$(echo "$RESPONSE" | jq '.queue_totals.messages_ready // 0') DELIVER_GET=$(echo "$RESPONSE" | jq '.message_stats.deliver_get // 0') echo "name=Custom Metrics|RabbitMQ|Total Messages, value=$MESSAGES" echo "name=Custom Metrics|RabbitMQ|Messages Ready, value=$MESSAGES_READY" echo "name=Custom Metrics|RabbitMQ|Deliver Get, value=$DELIVER_GET" 3.3 Integrate with AppDynamics Machine Agent Place the Script: Copy the script.sh script to the Machine Agent monitors directory: cp script.sh <MachineAgent_Dir>/monitors/RabbitMQMonitor/ 2. Create monitor.xml : Create a monitor.xml file to configure the Machine Agent: <monitor> <name>RabbitMQ</name> <type>managed</type> <enabled>true</enabled> <enable-override os-type="linux">true</enable-override> <description>RabbitMQ </description> <monitor-configuration> </monitor-configuration> <monitor-run-task> <execution-style>periodic</execution-style> <name>Run</name> <type>executable</type> <task-arguments> </task-arguments> <executable-task> <type>file</type> <file>script.sh</file> </executable-task> </monitor-run-task> </monitor> 3. Restart the Machine Agent: Restart the agent to apply the changes: cd <MachineAgent_Dir>/bin ./machine-agent & Step 4: Viewing Metrics in AppDynamics Log in to your AppDynamics Controller. Navigate to Servers > Custom Metrics. Look for metrics under: Custom Metrics|RabbitMQ You should see metrics like: Total Messages Messages Ready Deliver Get ... View more

How to Redirect Smart Agent Temporary Files to Avoid /tmp Space Limitations on Linux Installing any APM or Machine agent with Smart Agent on your Linux box will use /tmp directory to copy agent binaries before moving them to the intended directory. The problem You can get an error like below: error message = Error extracting Machine Agent in staging: reading file in zip archive: /tmp/.staging/machine-agent/jre/lib/modules: writing file: write /tmp/.staging/machine-agent/jre/lib/modules: no space left on device Error creating Machine Agent service: error installing service: error moving service file to destination: rename /tmp/.staging/appdynamics-machine-agent.service /etc/systemd/system/appdynamics-machine-agent.service: invalid cross-device link These errors are caused due to less space in /tmp folder or /tmp directory mounted on an external device. How to fix it You need to have Smart Agent use any other directory on our host then tmp. To do this: Go to <Smart-Agent-Home-Directory>, In my case the directory is /opt/appdynamics/appdsmartagent Commands in order: cd /opt/appdynamics/appdsmartagent ./smartagentctl stop export TMPDIR=/opt/appdynamics ./smartagentctl start Now in your logs, you will see {"severityText":"INFO","timestamp":"2024-10-04T16:30:29.692Z","name":"native","caller":"machine/task_helper.go:48","body":"downloaded file to ","downloaded file":"/opt/appdynamics/.staging/download/machineagent-bundle-64bit-linux-24.9.0.4408.zip"} {"severityText":"INFO","timestamp":"2024-10-04T16:30:29.692Z","name":"native","caller":"machine/task_helper.go:161","body":"Extracting zip","package.name":"8a5e85401b3a01ac5dadd6394c235dbf032ffa04;MACHINE_AGENT","src path":"/opt/appdynamics/.staging/download/machineagent-bundle-64bit-linux-24.9.0.4408.zip","dest path":"/opt/appdynamics/.staging/machine-agent"} This means, Smart Agent is now copying everything in /opt/appdynamics directory. ... View more

Currently, InfraViz doesn't let you deploy Custom extensions. If you wish to deploy custom extensions on Kubernetes using machine agents then this article is for you. This can be done in 2 ways: Creating a new Machine Agent Image Creating a new yaml file for Machine Agent Creating a new Machine Agent Image Now if you wish to use this method, which is modifying the Machine Agent image you need to take a step back and ask yourself: Do you need the extension on all nodes? If not, then if you deploy InfraViz in default by just updating the Image with extension then on the node where it works, everything will be fine but on others, you will have logs filled with ERROR/WARN messages which can potentially lead to Machine agent collector script timing out. Do you need a Machine Agent on all nodes? If not, then we are okay. We can use NodeSelector property of InfraViz and simply deploy this new Image using InfraViz on the specific node. In any case, the Dockerfile will look like below: FROM ubuntu:latest # Install curl and unzip RUN apt-get update && apt-get install -y curl unzip procps # Add and unzip the Machine Agent bundle ADD machineagent-bundle-64bit-linux-23.7.0.3689.zip /tmp/machineagent.zip RUN unzip /tmp/machineagent.zip -d /opt/appdynamics && rm /tmp/machineagent.zip # Set environment variable for Machine Agent home ENV MACHINE_AGENT_HOME /opt/appdynamics # Add AWS API Gateway Monitor and start-appdynamics script ADD create-open-file-extension-folder /opt/appdynamics/monitors ADD start-appdynamics ${MACHINE_AGENT_HOME} # Make start-appdynamics script executable RUN chmod 744 ${MACHINE_AGENT_HOME}/start-appdynamics # Set Java Home environment variable ENV JAVA_HOME /opt/appdynamics/jre/bin/java # Run AppDynamics Machine Agent CMD ["/opt/appdynamics/start-appdynamics"] NOTE: In the same directory as Dockerfile You need to have the appdynamics zip in your local. In my case, I have machineagent-bundle-64bit-linux-23.7.0.3689.zip in my local create-open-file-extension-folder is the extension folder which I am moving to /opt/appdynamics/monitors, This has my script.sh and monitor.xml file Remember for extensions, the Machine agent looks for folders and files in monitors directory. start-appdynamics.sh script. This is the content of start-appdynamics.sh script. You will need to edit it and add your Controller configuration. MA_PROPERTIES="-Dappdynamics.controller.hostName=xxx.saas.appdynamics.com" MA_PROPERTIES+=" -Dappdynamics.controller.port=443" MA_PROPERTIES+=" -Dappdynamics.agent.accountName=xxxx" MA_PROPERTIES+=" -Dappdynamics.agent.accountAccessKey=xx" MA_PROPERTIES+=" -Dappdynamics.controller.ssl.enabled=true" MA_PROPERTIES+=" -Dappdynamics.sim.enabled=true" MA_PROPERTIES+=" -Dappdynamics.docker.enabled=false" MA_PROPERTIES+=" -Dappdynamics.docker.container.containerIdAsHostId.enabled=true" # Start Machine Agent ${MACHINE_AGENT_HOME}/jre/bin/java ${MA_PROPERTIES} -jar ${MACHINE_AGENT_HOME}/machineagent.jar Great. Now all you need to do is build the Image and push it to your repository. Once done, in the InfraViz section, Update the Image section of InfraViz with this new Image Creating a new yaml file for Machine Agent Now the second option is using deployment and InfraViz together. I have created a infraviz-deployment.yaml file. This is a deployment that I am deploying on a specific node. apiVersion: apps/v1 kind: Deployment metadata: name: machine-agent-extension labels: app: machine-agent-extension spec: replicas: 1 selector: matchLabels: app: machine-agent-extension template: metadata: labels: app: machine-agent-extension spec: initContainers: - name: create-open-file-extension-folder image: busybox command: ['sh', '-c', 'mkdir -p /opt/appdynamics/monitors/open-file-extension && cp /tmp/config/* /opt/appdynamics/monitors/open-file-extension && chmod +x /opt/appdynamics/monitors/open-file-extension/script.sh'] volumeMounts: - name: config-volume mountPath: /tmp/config # Mount ConfigMap here temporarily - name: open-file-extension mountPath: /opt/appdynamics/monitors/open-file-extension # Target directory in emptyDir containers: - name: machine-agent-extension image: appdynamics/machine-agent:latest ports: - containerPort: 9090 env: - name: APPDYNAMICS_CONTROLLER_HOST_NAME value: "xxxx.saas.appdynamics.com" - name: APPDYNAMICS_CONTROLLER_PORT value: "443" - name: APPDYNAMICS_AGENT_ACCOUNT_NAME value: "xxx" - name: APPDYNAMICS_AGENT_ACCOUNT_ACCESS_KEY value: "xxx" - name: APPDYNAMICS_SIM_ENABLED value: "true" - name: APPDYNAMICS_CONTROLLER_SSL_ENABLED value: "true" volumeMounts: - name: open-file-extension mountPath: /opt/appdynamics/monitors/open-file-extension volumes: - name: config-volume configMap: name: open-file-extension-config # ConfigMap holding script.sh and monitor.xml - name: open-file-extension emptyDir: {} # EmptyDir to allow read/write nodeSelector: kubernetes.io/hostname: "ip-222-222-222-222.us-west-2.compute.internal" --- apiVersion: v1 kind: ConfigMap metadata: name: open-file-extension-config namespace: default data: script.sh: | #!/bin/bash # Get the current open files limit for the process open_files_limit=$(ulimit -n) ##Commentlineforcheck # Output the open files limit to stdout echo "name=Custom Metrics|OpenFilesLimitMonitor|OpenFilesLimit,value=$open_files_limit" monitor.xml: | <monitor> <name>OpenFile</name> <type>managed</type> <enabled>true</enabled> <enable-override os-type="linux">true</enable-override> <description>OpenFile</description> <monitor-configuration></monitor-configuration> <monitor-run-task> <execution-style>periodic</execution-style> <name>Run</name> <type>executable</type> <task-arguments></task-arguments> <executable-task> <type>file</type> <file>script.sh</file> </executable-task> </monitor-run-task> </monitor> Right now I am only monitoring one node though, so we will use InfraViz to monitor other nodes now. Taint the nodes that I don't want InfraViz to run now which is going to be above one ^ kubectl taint node ip-222-222-222-222.us-west-2.compute.internal machine-agent=false:NoSchedule Now, I can deploy InfraViz.yaml normally and it won't be deployed on ip-222 node. Now you have MA running on all of your nodes, one with extension, and the rest normally Please reach out to Support if you have any questions. ... View more

Step-by-Step Guide to Deploying AppDynamics Smart Agent Using Ansible on Linux Systems This article will guide you through installing the AppDynamics Smart Agent on a Linux system using Ansible. It covers downloading, configuring, and starting the Smart Agent in an automated fashion. This setup ensures that the Smart Agent is correctly configured for your environment. Prerequisites: Ansible Installed: Make sure Ansible is installed on the machine where you are running the playbook. Sudo Privileges: The playbook requires sudo (root) privileges to execute tasks. Download URL: You need a valid download URL for the AppDynamics Smart Agent. You can get this from the AppDynamics download site (replace the provided URL with your own). Steps to Set Up the Ansible Playbook 1. Directory Structure Create a directory structure for the Ansible playbook as follows: 2. Inventory File Define your target machine (localhost in this case) in the inventory file: [appd_agents] localhost ansible_connection=local 3. Ansible Playbook ( playbook.yml ) The main playbook references the smart_agent role. Ensure become: true is set to allow privilege escalation for necessary tasks: --- - hosts: appd_agents become: true roles: - smart_agent 4. Role Tasks ( roles/smart_agent/tasks/main.yml ) The tasks in this role will include downloading, unarchiving, configuring, and starting the Smart Agent. - name: Download AppDynamics Smart Agent using curl command: > curl -L -O -H "Authorization: Bearer <YOUR_AUTH_TOKEN>" "https://download.appdynamics.com/download/prox/download-file/appdsmartagent/<version>/appdsmartagent_64_linux_<version>.zip" args: chdir: /tmp - name: Unarchive the Smart Agent zip unarchive: src: /tmp/appdsmartagent_64_linux_<version>.zip dest: /opt/appdynamics/ remote_src: yes - name: Configure Smart Agent in config.ini replace: path: /opt/appdynamics/config.ini regexp: 'ControllerURL\s*=\s*.*' replace: 'ControllerURL=https://xxxxx.saas.appdynamics.com' - name: Set ControllerPort in config.ini replace: path: /opt/appdynamics/config.ini regexp: 'ControllerPort\s*=\s*.*' replace: 'ControllerPort=443' - name: Set FMServicePort in config.ini replace: path: /opt/appdynamics/config.ini regexp: 'FMServicePort\s*=\s*.*' replace: 'FMServicePort=443' - name: Set AccountAccessKey in config.ini replace: path: /opt/appdynamics/config.ini regexp: '^AccountAccessKey\s*=\s*.*' replace: 'AccountAccessKey=<YOUR_ACCOUNT_ACCESS_KEY>' - name: Ensure AccountName is set in the main section of config.ini lineinfile: path: /opt/appdynamics/config.ini regexp: '^AccountName\s*=' line: 'AccountName=xxxxx' insertafter: '^ControllerPort\s*=.*' - name: Enable SSL in config.ini replace: path: /opt/appdynamics/config.ini regexp: 'EnableSSL\s*=\s*.*' replace: 'EnableSSL=true' - name: Start Smart Agent shell: /opt/appdynamics/smartagentctl start --service > /tmp/log.log 2>&1 become: yes register: output 5. Replace the Download URL and other controller parameter Replace the download URL placeholder with your own Smart Agent download URL from the AppDynamics download site. In the command task for downloading the Smart Agent, replace <YOUR_AUTH_TOKEN> with your AppDynamics authentication token and replace <version> with the appropriate version for your Smart Agent. For example: - name: Download AppDynamics Smart Agent using curl command: > curl -L -O -H "Authorization: Bearer YOUR_AUTH_TOKEN" "https://download.appdynamics.com/download/prox/download-file/appdsmartagent/24.8.0.551/appdsmartagent_64_linux_24.8.0.551.zip" args: chdir: /tmp Replace Controller URL, Controller Port, AccessKey and AccountName with your credentials 6. Running the Playbook To run the playbook, execute the following command: ansible-playbook -i inventory playbook.yml Conclusion This Ansible playbook simplifies downloading, configuring, and running the AppDynamics Smart Agent on a Linux system. Make sure to replace the download URL and account details with your specific values, and you’ll have the agent up and running in no time. ... View more

Step-by-Step Guide to Setting Up AppDynamics Smart Agent on Windows Systems Go to the AppDynamics Downloads area in Accounts On the Agent tab, under the area "Type"  Select "Agent Management" and then download the AppDynamics Smart Agent for Windows For Windows, you require Administrator access to start Smart Agent. Therefore, you cannot start Smart Agent as a process in Windows Once you have downloaded the Smart Agent on Windows box, unzip the content The unzipped content will have the below files as of version 24.8.0 of AppDynamics Smart Agent You need to edit the config.ini file, specifically the below section: ControllerURL=<Your-AppDynamics-Controller-Url> ControllerPort=<Your-AppDynamics-Controller-port> FMServicePort=<Your-AppDynamics-Controller-port> AgentType = <Let-this-be-null> AccountAccessKey=<Your-AppDynamics-Controller-accessKey> AccountName=<Your-AppDynamics-Controller-Account-name> EnableSSL= <True-If-SSL-Is-Enabled-Else-False> Once this is edited, open the CMD prompt go the directory where SmartAgent is downloaded, and run: smartagentctl start --service​ Once this is done, SmartAgent should be installed as a service named “appdsmartagent”. You can confirm this from TaskManager.  Now, if you go to the AppDynamics Controller UI, under Agent Management -> Smart Agent, you will be able to see Smart Agent installed. ... View more

AppDynamics Cluster Agent allows you to auto-instrument your Applications running on Kubernetes. The auto-instrumentation injects APM agents on runtime, modifying your deployment spec with an init-container of AppDynamics APM agent. You can use different strategies to target Kubernetes deployments, StatefulSet, or DeploymentConfigs. In this article, we will cover instrumenting one deployment by using a label defined on the Deployment level (it will be the same for DeploymentConfig and StatefulSet) Let's take this forward with two sample deployments running in namespace abhi-java-apps-second. My first deployment: --- apiVersion: apps/v1 kind: Deployment metadata: name: tomcat-app-abhi labels: app: tomcat-app-abhi-second namespace: abhi-java-apps-second spec: replicas: 1 selector: matchLabels: app: tomcat-app-abhi template: metadata: labels: app: tomcat-app-abhi spec: containers: - name: tomcat-app-abhi #image: docker.io/abhimanyubajaj98/java-tomcat-sample-app-buildx:latest image: docker.io/abhimanyubajaj98/java-application:latest imagePullPolicy: Always ports: - containerPort: 8080 env: - name: JAVA_TOOL_OPTIONS value: -Xmx512m #- name: APPDYNAMICS_AGENT_UNIQUE_HOST_ID # value: $(cat /proc/self/cgroup | head -1 | awk -F '/' '{print $NF}' | cut -c 16-27) --- apiVersion: v1 kind: Service metadata: name: tomcat-app-service labels: app: tomcat-app-abhi namespace: abhi-java-apps-second spec: ports: - port: 8080 targetPort: 8080 selector: app: tomcat-app-abhi My Second deployment: --- apiVersion: apps/v1 kind: Deployment metadata: name: tomcat-app-abhi-labelmatch labels: app: tomcat-app-abhi-second-labelmatch namespace: abhi-java-apps-second spec: replicas: 1 selector: matchLabels: app: tomcat-app-abhi-labelmatch template: metadata: labels: app: tomcat-app-abhi-labelmatch spec: containers: - name: tomcat-app-abhi #image: docker.io/abhimanyubajaj98/java-tomcat-sample-app-buildx:latest image: docker.io/abhimanyubajaj98/java-application:latest imagePullPolicy: Always ports: - containerPort: 8080 env: - name: JAVA_TOOL_OPTIONS value: -Xmx512m #- name: APPDYNAMICS_AGENT_UNIQUE_HOST_ID # value: $(cat /proc/self/cgroup | head -1 | awk -F '/' '{print $NF}' | cut -c 16-27) --- apiVersion: v1 kind: Service metadata: name: tomcat-app-service-labelmatch labels: app: tomcat-app-abhi-labelmatch namespace: abhi-java-apps-second spec: ports: - port: 8080 targetPort: 8080 selector: app: tomcat-app-abhi-labelmatch Now, my use case is only for instrument deployment tomcat-app-abhi-labelmatch. To do this, I would need to edit my cluster-agent.yaml and add the below specs: instrumentationRules: - namespaceRegex: abhi-java-apps-second labelMatch: - app: tomcat-app-abhi-second-labelmatch tierName: abhiapps language: java imageInfo: image: "docker.io/appdynamics/java-agent:latest" agentMountPath: /opt/appdynamics imagePullPolicy: Always Now, after the deployment is done, only the deployment tomcat-app-abhi-labelmatch will have AppDynamics Java Agent. ... View more

Introduction In Kubernetes, securing communications between different components is essential for maintaining the integrity and confidentiality of your applications. Certificates play a pivotal role in this security ecosystem, enabling encrypted communication, and authentication, and ensuring that data remains tamper-proof. However, as Kubernetes environments grow in complexity, tracking which certificates are used by specific deployments can become challenging. This guide provides a comprehensive, secrets-focused approach to identifying and verifying the certificates used by a specific Kubernetes deployment, using the AppDynamics Cluster Agent as a practical example. Understanding Kubernetes Certificates What Are Certificates in Kubernetes? Certificates in Kubernetes are digital documents used to secure communications between different components, such as API servers, nodes, and deployed applications. These certificates are typically X.509 certificates and are used for encryption, authentication, and integrity checks. In Kubernetes, certificates can be issued by a Certificate Authority (CA) and are often managed automatically, although there are scenarios where manual intervention is required. What Are Secrets in Kubernetes? Kubernetes Secrets are objects used to store sensitive information, such as passwords, OAuth tokens, and, importantly, TLS certificates. Storing certificates as secrets allows Kubernetes to manage and distribute them securely to the pods that require them. Common Certificate Use Cases Some common uses of certificates in Kubernetes include: Securing API Server Communications: Ensuring that communications between the Kubernetes API server and nodes are encrypted. Ingress Controllers: Handling SSL/TLS termination for external traffic entering the cluster. Mutual TLS (mTLS) Between Services: Encrypting communication between microservices within the cluster. Application Routes: Securing external access to applications through routes or services. Understanding Kubernetes Secrets Types of Secrets: kubernetes.io/tls: Used specifically for storing TLS certificates. Opaque: A generic type of secret that can store any key-value pair, including certificates. kubernetes.io/dockercfg: Used for storing Docker configuration, particularly for private image registries. kubernetes.io/service-account-token: Used to manage service account tokens. Prerequisites Before you begin, ensure you have the following: kubectl: The command-line tool for interacting with Kubernetes clusters. OpenSSL: A tool for working with SSL/TLS certificates. Access: Sufficient permissions to access and manage secrets in the Kubernetes cluster. AppDynamics Cluster Agent installed: Install the Cluster Agent  Step-by-Step Guide to Identifying Certificates via Secrets Step 1: List All Secrets in the Namespace Certificates are stored within secrets. Start by listing all secrets in the relevant namespace to identify potential candidates. kubectl get secrets -n <namespace> In our case I have deployed Cluster agent in appdynamics namespace, So I will reference appdynamics in subsequent steps. Step 2: Filter Secrets Related to Certificates Kubernetes secrets that store TLS certificates usually have the type kubernetes.io/tls . Filter secrets by this type to narrow down your search. kubectl get secrets -n appdynamics -o json | jq '.items[] | select(.type=="kubernetes.io/tls") | .metadata.name' In AppDynamics, We don’t have a TLS secret by default so the above should return nothing. Step 3: Check Other Secrets If no kubernetes.io/tls secrets are found, inspect other secrets in the namespace to see if they contain certificates. This command will list all the secrets in the namespace, including those that might contain certificate data. kubectl get secrets -n appdynamics NAME TYPE DATA AGE appdynamics-cluster-agent-dockercfg-zvglk kubernetes.io/dockercfg 1 30m appdynamics-cluster-agent-token-5dt8s kubernetes.io/service-account-token 4 30m appdynamics-infraviz-dockercfg-f9ncg kubernetes.io/dockercfg 1 30m appdynamics-infraviz-token-5qpfs kubernetes.io/service-account-token 4 30m appdynamics-operator-dockercfg-v9k7k kubernetes.io/dockercfg 1 30m appdynamics-operator-token-478lb kubernetes.io/service-account-token 4 30m builder-dockercfg-wkdnm kubernetes.io/dockercfg 1 30m builder-token-nvjmq kubernetes.io/service-account-token 4 30m cluster-agent-secret Opaque 1 30m default-dockercfg-6dn5t kubernetes.io/dockercfg 1 30m default-token-m7hkp kubernetes.io/service-account-token 4 30m deployer-dockercfg-25ws4 kubernetes.io/dockercfg 1 30m deployer-token-dd9s9 kubernetes.io/service-account-token 4 30m sh.helm.release.v1.abhi-cluster-agent.v1 helm.sh/release.v1 1 30m Step 4: Inspect the Content of a Secret To examine the contents of a specific secret, such as appdynamics-cluster-agent-dockercfg-zvglk , use the following command: kubectl get secret appdynamics-cluster-agent-dockercfg-zvglk -n appdynamics -o yaml The output will look something like below kubectl get secret appdynamics-cluster-agent-token-5dt8s -n appdynamics -o yaml apiVersion: v1 data: ca.crt: xxxxxxxx namespace: YXBwZHluYW1pY3M= service-ca.crt: xxxxxx== token: xxxxxxx kind: Secret metadata: annotations: kubernetes.io/created-by: openshift.io/create-dockercfg-secrets kubernetes.io/service-account.name: appdynamics-cluster-agent kubernetes.io/service-account.uid: 582c10bc-b3fb-4435-9adb-7c6dfb25c2ff creationTimestamp: "2024-09-03T17:42:20Z" name: appdynamics-cluster-agent-token-5dt8s namespace: appdynamics resourceVersion: "84672" uid: a7cc1cba-27e4-4b46-a33d-212614af9cad type: kubernetes.io/service-account-token Step 5: Decode and Inspect the Certificate To check the details of the certificate (e.g., ca.crt ), decode it, and then use openssl to view its content: echo "Content of ca.crt" | base64 - decode | openssl x509 -noout -text If the certificate is found and properly decoded, the output will resemble: Certificate: Data: Version: 3 (0x2) Serial Number: ... Signature Algorithm: sha256WithRSAEncryption Issuer: OU=openshift, CN=kube-apiserver-lb-signer ... Conclusion This guide has walked you through identifying the certificates used by a specific Kubernetes deployment, focusing on secrets. By following these steps, you can effectively manage and verify the certificates that secure your Kubernetes environments, ensuring the integrity and confidentiality of your applications. ... View more

How to Encrypt and Secure Your Machine Agent AccessKey Encryption and securing your credentials are of utmost importance now. The Machine Agent lets you configure the encryption of your AccessKey. Let's go through the steps.   1. Navigate to <Ma-Home> directory and Let's create a keyStore with below command: jre/bin/java -jar lib/secure-credential-store-tool-1.3.23.jar generate_ks -filename '/opt/appdynamics/secretKeyStore' -storepass 'MyCredentialStorePassword' This will create the keyStore for you. The output should look like: Successfully created and initialized new KeyStore file: /opt/appdynamics/secretKeyStore 2. Let's create a password to access this keyStore: jre/bin/java -jar lib/secure-credential-store-tool-1.3.23.jar obfuscate -plaintext 'MyCredentialStorePassword' The output should look like: s_-001-12-oRQaGjKDTRs=xxxxxxxxxxxxx= 3. Encrypt your AccessKey: jre/bin/java -jar lib/secure-credential-store-tool-1.3.23.jar encrypt -filename /opt/appdynamics/secretKeyStore -storepass 'MyCredentialStorePassword' -plaintext 'xxxxxx' The output should look like: -001-24-mEZsR+xxxxxxxxx==xxxxxxxxxxxx== Great, now let's edit <MA-Home>/conf/controller-info.xml file and edit the accessKey while also adding a few more parameters for the encryption <account-access-key>-001-24-mEZsR+nrScSXlewlZbTQgg==xxxxxxxxx==</account-access-key> <credential-store-password>s_-001-12-xxxxxxx=xxxxxx=</credential-store-password> <credential-store-filename>/opt/appdynamics/secretKeyStore</credential-store-filename> <use-encrypted-credentials>true</use-encrypted-credentials> Great work! You can deploy your Machine Agent now! ... View more

Having the flexibility to use different AccessKey for different applications you auto-instrument with Cluster Agent is essential. This flexibility was introduced with the latest versions of the Cluster agent. Follow these steps: Un-instrument your application. Create a new secret with the accessKey with the below command: kubectl -n appdynamics create secret generic <secret-name> --from-literal=<custom-Controller-key-name>=<key-value>​ In this command <key-value> will be your AccessKey. For example: kubectl -n appdynamics create secret generic abhi-java-apps --from-literal=controller-key=xxxxx-fb91-4dfc-895a-xxxxx​ I modified my yaml and in the specific instrumentationRule section I added: - namespaceRegex: abhi-java-apps language: java matchString: tomcat-app-abhi-non-root appNameLabel: app runAsUser: 1001 runAsGroup: 1001 customSecretName: abhi-java-apps customSecretKey: controller-key imageInfo: image: "docker.io/appdynamics/java-agent:latest" agentMountPath: /opt/appdynamics imagePullPolicy: Always​ customSecretName is the secret name and customSecretKey is the Key for that secret. After this, I re-instrumented my application, and in the Cluster agent logs I confirmed [INFO]: 2024-08-19 14:45:09 - deploymenthandler.go:262 - custom secretName is %s and is %s %!(EXTRA string=abhi-java-apps, string=controller-key)​ Also, when I exec inside the application pod and grepped for env | grep -i Access, i confirmed that this AccessKey is used: wso2carb@tomcat-app-abhi-non-root-5d558dddf4-rllzc:/usr/local/tomcat/webapps$ env | grep -i Access APPDYNAMICS_AGENT_ACCOUNT_ACCESS_KEY=xxxxx-fb91-4dfc-895a-xxxxx JAVA_TOOL_OPTIONS=-Xmx512m -Dappdynamics.agent.accountAccessKey=xxxxx-fb91-4dfc-895a-xxxxx -Dappdynamics.agent.reuse.nodeName=true -Dappdynamics.socket.collection.bci.enable=true -javaagent:/opt/appdynamics-java/javaagent.jar​ Additional Resources Use Custom Access Key ... View more

Configuring JVM Options to Increase Metric Limits in Machine Agent By default, the Machine agent posts 450 metrics every minute. However, this limit can easily get hit when you configure custom extensions, you have a huge number of processes running on your VM or your OS has increased churn. To fix this, we need to add below JVM  -Dappdynamics.agent.maxMetrics=1000 This JVM will increase the metrics the Machine agent will publish on every POST to the controller. To pass this on to Linux you can simply start a Machine agent with this JVM. For example: java -Dappdynamics.agent.maxMetrics=2500 -jar machineagent.jar On Windows, you can edit the MachineAgentService.vmoptions file located in <MA-Home>/bin folder and restart the Machine Agent. ... View more