Step-by-Step Guide to Verifying SSL Certificates Using AppDynamics InfraViz In this article, you will learn how to verify and resolve SSL/TLS certificate trust issues for an On-Premises Controller when deploying AppDynamics Infrastructure Visibility (InfraViz). Below are the step-by-step instructions to create the necessary Kubernetes secrets, configure infraviz.yaml, and verify the certificate mount in your cluster. Step 1: Create an Infrastructure Visibility Secret using Keystore Credentials a) Download the SSL Certificate from the Controller Fetch the certificate chain directly from the Controller host (this command works in environments without an intervening proxy): keytool -printcert -sslserver <appdynamics-controller-host>:443 -rfc > custom-ssl.pem   b) Import the CA Certificate into a Java Keystore (cacerts.jks) Import the extracted custom-ssl.pem file into a Java Keystore: keytool -import -alias rootCA -file custom-ssl.pem -keystore cacerts.jks -storepass <your-password> c) Create the Keystore File Secret Create the Kubernetes secret containing the cacerts.jks file: Direct command: kubectl -n appdynamics create secret generic infraviz-ssl-cert --from-file=cacerts.jks Generate a YAML manifest file: kubectl -n appdynamics create secret generic infraviz-ssl-cert --from-file=cacerts.jks --dry-run=client -o yaml > infraviz_ssl-cert.yaml d) Create the Keystore Password Secret Create the Kubernetes secret containing the keystore password: Direct command: kubectl -n appdynamics create secret generic infraviz-ssl-cert-pass --from-literal=keystore-password="<your-password>" Generate a YAML manifest file: kubectl -n appdynamics create secret generic infraviz-ssl-cert-pass --from-literal=keystore-password="<your-password>" --dry-run=client -o yaml > infraviz_ssl-cert_pass.yaml Verification: Verify Secrets in the Namespace Ensure that both secrets are present and contain the correct data keys: kubectl get secret -n appdynamics Output: NAME TYPE DATA AGE cluster-agent-secret Opaque 1 34m infraviz-ssl-cert Opaque 1 41s infraviz-ssl-cert-pass Opaque 1 41s Describe each secret to verify the contents: kubectl describe secret infraviz-ssl-cert -n appdynamics Output: Name: infraviz-ssl-cert Namespace: appdynamics Labels: <none> Annotations: <none> Type: Opaque Data ==== cacerts.jks: 2134 bytes   kubectl describe secret infraviz-ssl-cert-pass -n appdynamics Output: Name: infraviz-ssl-cert-pass Namespace: appdynamics Labels: <none> Annotations: <none> Type: Opaque Data ==== keystore-password: 6 bytes   Step 2: Update the infraviz.yaml Configuration File Specify keyStoreFileSecret and keyStorePasswordSecret in the infraviz.yaml configuration file to apply the custom SSL truststore. Key parameters: keyStoreFileSecret: infraviz-ssl-cert keyStorePasswordSecret: infraviz-ssl-cert-pass Example infraviz.yaml: apiVersion: v1 kind: ServiceAccount metadata: name: appdynamics-infraviz namespace: appdynamics --- apiVersion: cluster.appdynamics.com/v1alpha1 kind: InfraViz metadata: name: appdynamics-infraviz namespace: appdynamics spec: controllerUrl: "https://<appdynamics-controller-host>:443" image: "<your-docker-registry>/appdynamics/machine-agent:latest" account: "<account-name>" enableContainerHostId: true keyStoreFileSecret: infraviz-ssl-cert keyStorePasswordSecret: infraviz-ssl-cert-pass Apply the updated configuration: kubectl apply -f infraviz.yaml Step 3: Check the InfraViz Pod Description Verify that the secret volume and volume mount are correctly injected into the InfraViz pod: kubectl describe pod/<infraviz-pod-name> -n appdynamics a) Check Volume Mounts: Mounts: /hostroot/ from appd-host-volume (ro) /opt/appdynamics/conf/cacerts.jks from agent-ssl-cert (rw,path="cacerts.jks") /opt/appdynamics/conf/logging from ma-log-volume (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-689km (ro) b) Check Volumes: Volumes: ma-log-volume: Type: ConfigMap (a volume populated by a ConfigMap) Name: ma-log-config Optional: false appd-host-volume: Type: HostPath (bare host directory volume) Path: / HostPathType: Directory agent-ssl-cert: Type: Secret (a volume populated by a Secret) SecretName: infraviz-ssl-cert Optional: false Step 4: Verify the Presence of cacerts.jks in the InfraViz Container Access the running InfraViz container shell and confirm that the cacerts.jks file is present in the configuration directory /opt/appdynamics/conf: kubectl exec -it <infraviz-pod-name> -n appdynamics -- /bin/sh Inside the container shell: bash-5.2$ cd /opt/appdynamics/conf bash-5.2$ pwd /opt/appdynamics/conf bash-5.2$ ls cacerts.jks controller-info.xml controller-info.xml.asc logging otel prometheus I hope this article was helpful! ... View more

How to enable DEBUG Level Logging for AppDynamics Smart Agent This article explains how to enable DEBUG level logging for the AppDynamics Smart Agent by updating the config.ini file and restarting the Smart Agent service. Why do I need to collect the debug-level log file? The Smart Agent logs entries at the info level by default. Sometimes, debug-level logs are required to investigate an issue in more detail. Debug-level logging provides more detailed and insightful entries, which can help identify the root cause of the experienced issue. These logs are especially useful when standard info logs do not contain enough detail for troubleshooting. You can collect Smart Agent logs from the server where the Smart Agent is installed. 1. Locate the Smart Agent configuration file The Smart Agent configuration file is located in the Smart Agent installation directory, for example: /opt/appdynamics/appdsmartagent/config.ini In your environment, the relevant section currently looks like this: [Telemetry] LogLevel = info LogFile = log.log 2. Change the logging level to DEBUG Edit the config.ini file and change the telemetry logging level from: [Telemetry] LogLevel = info to: [Telemetry] LogLevel = debug After the change, the section should look like this: [Telemetry] LogLevel = debug LogFile = log.log 3. Confirm the log file location The Smart Agent writes logs to the file configured by the LogFile parameter. Default value: LogFile = log.log In your example, the default log file is: /opt/appdynamics/appdsmartagent/log.log 4. Restart the Smart Agent service After saving the configuration change, restart the Smart Agent service: sudo systemctl restart smartagent 5. Verify DEBUG logging After the restart, check the configured log file: /opt/appdynamics/appdsmartagent/log.log You can use the following command: tail -f /opt/appdynamics/appdsmartagent/log.log Confirm that debug-level entries are being written to the log file. 6. Collect debug-level logs from the requested Smart Agent action Reproduce or perform the requested Smart Agent action while debug-level logging is enabled. This ensures the log file captures the detailed entries needed for troubleshooting. 7. Provide the log file for troubleshooting If Smart Agent logs are requested for troubleshooting, provide the configured log file. Default log file: /opt/appdynamics/appdsmartagent/log.log   ... View more

Why do I need to collect the debug-level startup logs?   By default, the InfraViz 'Machine Agent' logs entries at the INFO level. Since the container maintains a maximum of five log files, you must restart the InfraViz pod to capture startup logs. Debug-level logs are necessary to investigate this issue, as they provide the detailed information required to identify the root cause. There are two ways to collect these agent log files at the desired logging level By editing the currently running InfraViz By updating the InfraViz with a new configuration. 1. Edit the currently running InfraViz Determine the infraviz object name: kubectl get infraviz -n appdynamics Edit the YAML file as shown below to set the logLevel to DEBUG, then save it (similar to using the vi editor): kubectl edit infraviz <infraviz_name> -n appdynamics Edit the file to include:                             logLevel: debug Save the configuration: `:wq` Now InfraViz should spin up a new pod. Keep the InfraViz running for 15 minutes in debug logging mode. Collect logs by executing: kubectl cp appdynamics/<infraviz_pod_name>:/opt/appdynamics/logs -c appd-infra-agent ./logs Revert the changes 2. Update the Cluster Agent with a new configuration. Edit the configuration: a) Kubernetes CLI installation -> "infraviz.yaml" file         logLevel: debug b) Helm Chart installation -> "values.yaml" file # InfraViz config      infraViz:           logProperties:                  logLevel: debug Apply the changes a) Kubernetes CLI installation -> kubectl apply -f infraviz.yaml b) Helm Chart installation -> helm upgrade <release_name> -f values.yaml appdynamics-cloud-helmcharts/cluster-agent -n appdynamics Now the InfraViz should spin up a new cluster agent pod. Keep the InfraViz running for 5 minutes in debug logging mode. Collect logs by executing: kubectl cp appdynamics/<infraviz_pod_name>:/opt/appdynamics/logs -c appd-infra-agent ./logs Revert the changes   I hope this article was helpful. Feel free to ask in case of any questions. ... View more

Why do I need to collect the debug-level log file? The Cluster Agent by default logs the entries at the info level. Sometimes, the debug-level log files are necessary to investigate an experienced issue. Debug-level logging logs are more insightful entries that can be later used to identify the root cause of the experienced issue. There are two ways you can collect the agent log files at the desired logging level. By editing the currently running Cluster Agent By updating the Cluster Agent with a new configuration. 1. Edit the currently running Cluster Agent Determine the clusteragent object name: kubectl -n appdynamics edit clusteragent -n appdynamics Edit the YAML file as shown below to set the logLevel to DEBUG, then save it (similar to using the vi editor): kubectl edit clusteragent <cluster_agent_name> -n appdynamics Edit the file to include: Save the configuration: `:wq` Delete Cluster Agent pod with old configuration: kubectl delete pod <cluster-agent-pod> -n appdynamics Now cluster should spin up a new cluster agent pod. Keep the cluster-agent running for 15 minutes in debug logging mode. Collect logs by executing: kubectl logs <cluster-agent-pod> -n appdynamics > cluster-agent.log Revert the changes 2. Update the Cluster Agent with a new configuration. Edit the configuration: a) Kubernetes CLI installation -> "cluster-agent.yaml" file b) Helm Chart installation -> "values.yaml" file Apply the changes a) Kubernetes CLI installation -> kubectl apply -f cluster-agent.yaml b) Helm Chart installation -> helm upgrade <release_name> -f values.yaml appdynamics-cloud-helmcharts/cluster-agent -n appdynamics Now cluster should spin up a new cluster agent pod. Keep the cluster-agent running for 15 minutes in debug logging mode. Collect logs by executing: kubectl logs <cluster-agent-pod> -n appdynamics > cluster-agent.log Revert the changes   I hope this article was helpful. Feel free to ask in case of any questions. ... View more

Step-by-Step Guide to Verifying SSL Certificates Using AppDynamics Cluster Agent with Target Allocator In this article, you will learn how to check if there are issues with the SSL Certificate for the On-Premises Controller using the Cluster Agent with Target Allocator. Below are the steps you can independently verify in the cluster. Step-by-Step Guide:   Secret Check if a Secret containing the certificate named "custom-ssl.pem" is created. Cluster Agent Check if the Cluster Agent configuration file contains a parameter with the name of the Secret containing the certificate: "cluster-agent.yaml" -> Kubernetes CLI "values.yaml" -> Helm Chart Check if the Cluster Agent pod description contains: kubectl describe pod <cluster_agent_pod_name> -n appdynamics Environment variable "APPDYNAMICS_CUSTOM_SSL_SECRET: ssl-cert" Volume mount Volume "agent-ssl-cert" Check the presence of "/opt/appdynamics/ssl/custom-ssl.pem" in the Cluster Agent container. kubectl exec -it <cluster_agent_pod_name> -n appdynamics -- /bin/sh   Target Allocator Check if the Target Allocator pod description contains: kubectl describe pod <target_allocator_pod_name> -n appdynamics Environment variable "APPDYNAMICS_CUSTOM_SSL_SECRET: ssl-cert" Volume mount Volume "agent-ssl-cert" Check the presence of "/opt/appdynamics/ssl/custom-ssl.pem" in the Target Allocator container. kubectl exec -it <target_allocator_pod_name> -n appdynamics -- /bin/sh   I hope this article was helpful. Let me know in the comments if you have any questions or if something isn't working.  ... View more

Resolving AppDynamics Auto-Instrumentation Challenges in Capsule Tenant This article provides a step-by-step guide to replicating a local environment using Capsule, focusing on resolving common issues with AppDynamics auto-instrumentation within Kubernetes namespaces. It covers the installation of Capsule, tenant creation, and namespace setup, followed by troubleshooting symptoms, diagnosis, and solutions. The guide is designed for technical community members seeking to optimize their Capsule and AppDynamics integration. How to replicate locally Install Capsule https://github.com/projectcapsule/capsule/blob/main/charts/capsule/README.md Follow the instructions to create your first tenant https://capsule.clastix.io/docs/general/getting-started#login-as-tenant-owner Create the "appdynamics" and "oil-production" namespaces; they will be attached to the same Capsule Tenant. Symptoms The Cluster Agent cannot auto-instrument any application. Diagnosis There are no AppDynamics annotations in the Deployment/StatefulSet description.   There is no AppDynamics annotation in the namespace description.   In the Cluster Agent debug logs, many errors are related to the Cluster and Capsule tenant configuration. A single ERROR and 'Adding instrumentation task' are repeated every 5 minutes. The same situation occurs in the customer environment. [ERROR]: 2025-04-03 08:56:14 - utils.go:1086 - Unable to update namespace oil-production after 4 retries [DEBUG]: 2025-04-03 08:56:14 - deploymenthandler.go:623 - Adding instrumentation task for deployment oil-production/java-app-to-instrument [DEBUG]: 2025-04-03 08:56:14 - instrumentationconfig.go:794 - rule default|java|oil-production matches Deployment java-app-to-instrument in namespace oil-production with labels map[appd_agent:enable] [DEBUG]: 2025-04-03 08:56:14 - instrumentationconfig.go:805 - Found a matching rule {default|java|oil-production .* map[appd_agent:[enable]] java first JAVA_TOOL_OPTIONS map[agent-mount-path:/opt/appdynamics alpine-init-container-image: image:docker.io/appdynamics/java-agent:25.2.0 image-pull-policy:Always] map[bci-enabled:true port:3892] 0 0 false false false false map[] map[] map[] map[] 0 false [] true} for Deployment java-app-to-instrument in namespace oil-production with labels map[appd_agent:enable] [DEBUG]: 2025-04-03 08:56:14 - instrumentationconfig.go:338 - Instrumentation state for Deployment java-app-to-instrument in namespace oil-production with labels map[appd_agent:enable] is true​   Solution Cluster Agent service accounts require additional privileges in the Capsule Tenant: Original Tenant Tenant with additional privileges for Cluster Agent service accounts. All roles available here:  https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles I hope this article was helpful ... View more

Step-by-Step Guide to Verifying SSL Certificates Using AppDynamics Cluster Agent In this article, you will learn how to check if there are issues with the SSL Certificate for the On-Premises Controller using the Cluster Agent. Below are the steps you can independently verify in the cluster.   Next steps: Check if a Secret containing the certificate named "custom-ssl.pem" is created. Check if the Cluster Agent configuration file contains a parameter with the name of the Secret containing the certificate: "cluster-agent.yaml" -> Kubernetes CLI "values.yaml" -> Helm Chart Check if the Cluster Agent pod description contains: kubectl describe pod <cluster_agent_pod_name> -n appdynamics Environment variable "APPDYNAMICS_CUSTOM_SSL_SECRET: ssl-cert" Volume mount Volume "agent-ssl-cert" Check the presence of "/opt/appdynamics/ssl/custom-ssl.pem" in the Cluster Agent container. kubectl exec -it <cluster_agent_pod_name> -n appdynamics -- /bin/sh I hope this article was helpful  ... View more

What does this error mean? We usually observe the log message below in the Application startup logs when the agent is unable to connect with the controller to retrieve the nodeName (in the case of using reuse.nodeName).  Started AppDynamics Java Agent Successfully. [Thread-0] Tue Apr 02 09:46:04 UTC 2019[INFO]: JavaAgent - Started AppDynamics Java Agent Successfully. 2019-04-02 09:46:09,545 ERROR Recursive call to appender Buffer 2019-04-02 09:46:09,547 ERROR Recursive call to appender Buffer Next steps: Could you please check if any logs are generated under the /opt/appdynamics-java/ver.xxx.xx/logs/  directory and share them if available? If there are no logs, please add the configuration line below under the instrumentatioRules applied to the problematic pod:" customAgentConfig: -Dappdynamics.agent.reuse.nodeName=false -Dappdynamics.agent.nodeName=test   If you are using Cluster Agent version >=23.11.0, to force re-instrumentation, you need to use the additional parameter in the default auto-instrumentation properties: enableForceReInstrumentation: true   apiVersion: cluster.appdynamics.com/v1alpha1 kind: Clusteragent metadata: name: k8s-cluster-agent namespace: appdynamics spec: # cluster agent properties # ... # required to enable auto-instrumentation instrumentationMethod: Env # default auto-instrumentation properties # may be overridden in an instrumentationRule containerAppCorrelationMethod: proxy nsToInstrumentRegex: default defaultAppName: "" enableForceReInstrumentation: true # ADDED # ... # one or more instrumentationRules instrumentationRules: - namespaceRegex: default customAgentConfig: -Dappdynamics.agent.reuse.nodeName=false -Dappdynamics.agent.nodeName=test # ADDED imageInfo: image: "docker.io/appdynamics/java-agent:24.8.1" agentMountPath: /opt/appdynamics imagePullPolicy: Always Afterward, please apply the changes and wait for the cluster agent to implement the new instrumentation. Then, collect the agent logs from the /opt/appdynamics-java/ver.xxx.xx/logs/ directory and attach them to the ticket. How do you collect logs from a Kubernetes pod? 1. Enter the container and pack the agent logs into a tar file. kubectl exec -it pod <pod_name> -- bash cd /opt/appdynamics-java/ver24.x.x.x/logs/ tar -cvf /java-agent-logs.tar test 2. Copy the created tar file. kubectl cp <some-namespace>/<some-pod>:/java-agent-logs.tar ./java-agent-logs.tar I hope this article was helpful/ Łukasz Kociuba ... View more