topic Re: Auto instrument not working in Splunk AppDynamics

Auto instrument not working

sattha_puangput — Mon, 28 Jun 2021 16:38:02 GMT

Hi All,

Currently, I try to implement cluster-operator and cluster-agent on K8s. the cluster metric shows under the "Server/Cluster" menu correctly. However, I would like to config auto-instrument for Java application but no hope follow the validate guide here is still not much helpful...

https://docs.appdynamics.com/21.5/en/infrastructure-visibility/monitor-kubernetes-with-the-cluster-agent/auto-instrument-applications-with-the-cluster-agent/validate-auto-instrumentation#ValidateAuto-Instrumentation-TroubleshootAuto-InstrumentationWhenNotApplied

apiVersion: appdynamics.com/v1alpha1
kind: Clusteragent
metadata:
  name: k8s-cluster-agent
  namespace: appdynamics
spec:
  appName: "TEST-CLUSTER"
  controllerUrl: "https://xxx.appdynamics.com"
  account: "xxxdev"
  logLevel: DEBUG
  logFileSizeMb: 7
  logFileBackups: 6
  # docker image info
  image: "my.own.repository.org:5000/appdynamics/cluster-agent:latest"
  serviceAccountName: appdynamics-cluster-agent
  nsToMonitor:
    - "appdynamics"
    - "cma-dev"
    - "cma-qa"
  #
  # auto-instrumentation config
  #
  instrumentationMethod: Env
  nsToInstrumentRegex: "xxx-dev|xxx-qa"
  appNameStrategy: label
  defaultAppName: not-specific
  defaultCustomConfig: "-Dappdynamics.agent.maxMetrics=15000"
  defaultEnv: JAVA_TOOL_OPTIONS
  resourcesToInstrument:
    - Deployment
  imageInfo:
    java:
      image: "my.own.repository.org:5000/appdynamics/java-agent:latest"
      agentMountPath: /opt/appdynamics
  instrumentationRules:
    - namespaceRegex: "xxx-dev"
      language: java
      env: JAVA_TOOL_OPTIONS
      appNameLabel: app
      instrumentContainer: first
      imageInfo:
        image: "my.own.repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics
    - namespaceRegex: "xxx-qa"
      language: java
      env: JAVA_TOOL_OPTIONS
      appNameLabel: app
      instrumentContainer: first
      imageInfo:
        image: "my.own.repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics

But I got this....

[DEBUG]: 2021-06-28 16:12:36 - instrumentationconfig.go:628 - rule xxx-qa matches Deployment spring-api-productprofile in namespace xxx-qa with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-28 16:12:36 - instrumentationconfig.go:639 - Found a matching rule {xxx-qa  map[]   java first  -Dappdynamics.agent.maxMetrics=15000 JAVA_TOOL_OPTIONS map[agent-mount-path:/opt/appdynamics image:my-own-repository.org:5000/appdynamics/java-agent:latest] map[bci-enabled:true port:3892] 0 0 app  0 false []} for Deployment spring-api-productprofile in namespace xxx-qa with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-28 16:12:36 - instrumentationconfig.go:249 - Instrumentation state for Deployment spring-api-productprofile in namespace xxx-qa with labels map[app:spring-api-productprofile] is false
[DEBUG]: 2021-06-28 16:12:36 - instrumentationconfig.go:628 - rule xxx-dev matches Deployment spring-api-deliveries in namespace xxx-dev with labels map[app:spring-api-deliveries]
[DEBUG]: 2021-06-28 16:12:36 - instrumentationconfig.go:639 - Found a matching rule {xxx-dev  map[]   java first  -Dappdynamics.agent.maxMetrics=15000 JAVA_TOOL_OPTIONS map[agent-mount-path:/opt/appdynamics image:my-own-repository.org:5000/appdynamics/java-agent:latest] map[bci-enabled:true port:3892] 0 0 app  0 false []} for Deployment spring-api-deliveries in namespace xxx-dev with labels map[app:spring-api-deliveries]
[DEBUG]: 2021-06-28 16:12:36 - instrumentationconfig.go:249 - Instrumentation state for Deployment spring-api-deliveries in namespace xxx-dev with labels map[app:spring-api-deliveries] is false

Please advice

Re: Auto instrument not working

Morelz — Tue, 29 Jun 2021 13:05:37 GMT

Hi Sattha

You have set the appNameStrategy : label

And in the instrumentation rules, you state that it should use the below label of the Java pods to name the Application.

Do you have a label in your java app's deployment spec that is named "app"? As it will look for that label to set the Application Name. If it's empty it wont register. Your logs show that the instrumentation rule is matching the pod so the rest seem to be correct.

appNameLabel: app

I would suggest to set the following, to just validate testing, before switching back to using labels to set the AppD Application Name

appNameStrategy: manual

Change

appNameLabel: app to appName: XXX_DEV

appName: XXX_QA

Then redeploy and let us know

Ciao

Re: Auto instrument not working

sattha_puangput — Wed, 30 Jun 2021 01:21:52 GMT

Thank you for your advice, I update configure to this one now... but still got the same log output

apiVersion: appdynamics.com/v1alpha1
kind: Clusteragent
metadata:
  name: k8s-cluster-agent
  namespace: appdynamics
spec:
  appName: "SEACMACLUSTER-TH-DEVQA"
  controllerUrl: "https://xxx.saas.appdynamics.com"
  account: "xxx"
  logLevel: DEBUG
  logFileSizeMb: 7
  logFileBackups: 6
  # docker image info
  image: "my-own-repository.org:5000/appdynamics/cluster-agent:latest"
  serviceAccountName: appdynamics-cluster-agent
  nsToMonitor:
    - "appdynamics"
    - "cma-dev"
    - "cma-qa"
  proxyUrl: "http://xx.xx.xx.xx:80"
  proxyUser: "xxx\\svc_localapp_dev"
  #
  # auto-instrumentation config
  #
  instrumentationMethod: Env
  nsToInstrumentRegex: cma-dev|cma-qa
  appNameStrategy: manual
  defaultAppName: SEACMA-TH-NONPROD
  #defaultCustomConfig: "-Dappdynamics.agent.maxMetrics=15000"
  defaultEnv: JAVA_TOOL_OPTIONS
  resourcesToInstrument:
    - Deployment
  instrumentContainer: first
  imageInfo:
    java:
      image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
      agentMountPath: /opt/appdynamics
    dotnetcore:
      image: "docker.io/appdynamics/dotnet-core-agent:latest"
      agentMountPath: /opt/appdynamics
    nodejs:
      image: "docker.io/appdynamics/nodejs-agent:20.8.0-stretch-slimv10"
      agentMountPath: /opt/appdynamics
  instrumentationRules:
    - namespaceRegex: "cma-dev"
      appName: SEACMA-TH-DEV
      env: JAVA_TOOL_OPTIONS
      instrumentContainer: first
      imageInfo:
        image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics
    - namespaceRegex: "cma-qa"
      appName: SEACMA-TH-QA
      env: JAVA_TOOL_OPTIONS
      instrumentContainer: first
      imageInfo:
        image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics

Log

[DEBUG]: 2021-06-30 01:08:59 - instrumentationconfig.go:628 - rule cma-qa matches Deployment spring-api-productprofile in namespace cma-qa with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-30 01:08:59 - instrumentationconfig.go:639 - Found a matching rule {cma-qa  map[] SEACMA-TH-QA  java first   JAVA_TOOL_OPTIONS map[agent-mount-path:/opt/appdynamics image:my-own-repository:5000/appdynamics/java-agent:latest] map[bci-enabled:true port:3892] 0 0   0 false []} for Deployment spring-api-productprofile in namespace cma-qa with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-30 01:08:59 - instrumentationconfig.go:249 - Instrumentation state for Deployment spring-api-productprofile in namespace cma-qa with labels map[app:spring-api-productprofile] is false


[DEBUG]: 2021-06-30 01:08:59 - instrumentationconfig.go:628 - rule cma-dev matches Deployment spring-api-productprofile in namespace cma-dev with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-30 01:08:59 - instrumentationconfig.go:639 - Found a matching rule {cma-dev  map[] SEACMA-TH-DEV  java first   JAVA_TOOL_OPTIONS map[agent-mount-path:/opt/appdynamics image:my-own-repository.org:5000/appdynamics/java-agent:latest] map[bci-enabled:true port:3892] 0 0   0 false []} for Deployment spring-api-productprofile in namespace cma-dev with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-30 01:08:59 - instrumentationconfig.go:249 - Instrumentation state for Deployment spring-api-productprofile in namespace cma-dev with labels map[app:spring-api-productprofile] is false

What I try...

check K8s deployment config to make sure that config is applied to the current "k8s-cluster-agent" pod
delete and recreate application deployment but no init-container config apply (it should not I think since Instrument still false

Current dashboard screenshot...

clusterapplication

Re: Auto instrument not working

Morelz — Wed, 30 Jun 2021 06:58:01 GMT

Thanks Sattha

Could you remove all instances of the following from the cluster agent yaml

instrumentContainer: first

and add the following to each of the 2 instrumentation rules

containerMatchString: .*

The redeploy, and check if it instruments the pods now

Re: Auto instrument not working

sattha_puangput — Wed, 30 Jun 2021 16:22:46 GMT

I try to add the following config but it does not work...

instrumentContainer: select
containerMatchString: .*

apiVersion: appdynamics.com/v1alpha1
kind: Clusteragent
metadata:
  name: k8s-cluster-agent
  namespace: appdynamics
spec:
  appName: "SEACMACLUSTER-TH-DEVQA"
  controllerUrl: "https://xxxxxdev.saas.appdynamics.com"
  account: "xxxxxdev"
  logLevel: DEBUG
  logFileSizeMb: 7
  logFileBackups: 6
  # docker image info
  image: "my-own-repository.org:5000/appdynamics/cluster-agent:latest"
  serviceAccountName: appdynamics-cluster-agent
  ### Uncomment the following two lines if you need pull secrets
  #imagePullSecrets:
  #  name: "<your-docker-pull-secret-name>"
  nsToMonitor:
    - "appdynamics"
    - "cma-dev"
    - "cma-qa"
  proxyUrl: "http://172.30.60.46:80"
  proxyUser: "Th-xxxxx\\svc_localapp_dev"
  #
  # auto-instrumentation config
  #
  instrumentationMethod: Env
  nsToInstrumentRegex: cma-dev|cma-qa
  appNameStrategy: manual
  defaultAppName: SEACMA-TH-NONPROD
  instrumentContainer: select
  #defaultCustomConfig: "-Dappdynamics.agent.maxMetrics=15000"
  defaultEnv: JAVA_TOOL_OPTIONS
  resourcesToInstrument:
    - Deployment
  imageInfo:
    java:
      image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
      agentMountPath: /opt/appdynamics
    dotnetcore:
      image: "docker.io/appdynamics/dotnet-core-agent:latest"
      agentMountPath: /opt/appdynamics
    nodejs:
      image: "docker.io/appdynamics/nodejs-agent:20.8.0-stretch-slimv10"
      agentMountPath: /opt/appdynamics
  instrumentationRules:
    - namespaceRegex: "cma-dev"
      appName: SEACMA-TH-DEV
      env: JAVA_TOOL_OPTIONS
      instrumentContainer: select
      containerMatchString: .*
      imageInfo:
        image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics
    - namespaceRegex: "cma-qa"
      appName: SEACMA-TH-QA
      env: JAVA_TOOL_OPTIONS
      instrumentContainer: select
      containerMatchString: .*
      imageInfo:
        image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics

Still same error

[DEBUG]: 2021-06-30 16:12:11 - instrumentationconfig.go:628 - rule cma-qa matches Deployment spring-api-productprofile in namespace cma-qa with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-30 16:12:11 - instrumentationconfig.go:639 - Found a matching rule {cma-qa  map[] SEACMA-TH-QA  java select .*  JAVA_TOOL_OPTIONS map[agent-mount-path:/opt/appdynamics image:pvnexusho001th.dmz.th-tesco.org:5000/appdynamics/java-agent:latest] map[bci-enabled:true port:3892] 0 0   0 false []} for Deployment spring-api-productprofile in namespace cma-qa with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-30 16:12:11 - instrumentationconfig.go:249 - Instrumentation state for Deployment spring-api-productprofile in namespace cma-qa with labels map[app:spring-api-productprofile] is false

What I try

Update config according to your suggestion
Delete cluster-operator K8S deployment
Delete cluster-agent K8S deployment
recreate both cluster-operator and cluster-agent
check deployment configuration change to what we need
delete application deployment and recreate it --- not thing happen no init-container

Re: Auto instrument not working

Morelz — Thu, 01 Jul 2021 07:40:44 GMT

Hi Sattha

Can you try the below YAML Config

apiVersion: appdynamics.com/v1alpha1
kind: Clusteragent
metadata:
  name: k8s-cluster-agent
  namespace: appdynamics
spec:
  appName: "SEACMACLUSTER-TH-DEVQA"
  controllerUrl: "https://xxxxxdev.saas.appdynamics.com"
  account: "xxxxxdev"
  image: "my-own-repository.org:5000/appdynamics/cluster-agent:latest"
  logLevel: Debug
  logFileSizeMb: 10
  logFileBackups: 5
  stdoutLogging: "true"
  serviceAccountName: appdynamics-cluster-agent
  nsToMonitorRegex: appdynamics|cma-dev|cma-qa
  proxyUrl: "http://172.30.60.46:80"
  proxyUser: "Th-xxxxx\\svc_localapp_dev"
  #
  # auto-instrumentation config
  #
  instrumentationMethod: Env
  nsToInstrumentRegex: cma-dev|cma-qa
  appNameStrategy: manual
  defaultAppName: SEACMA-TH-NONPROD
  defaultEnv: JAVA_TOOL_OPTIONS
  instrumentationRules:
    - namespaceRegex: cma-dev
      language: java
      appName: SEACMA-TH-DEV
      containerMatchString: .*
      imageInfo:
        image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics

Re: Auto instrument not working

sattha_puangput — Thu, 01 Jul 2021 14:40:35 GMT

I try your .yaml file but when creating `cluster-agent.yaml` there is the following error

error: error validating "appdynamics/cluster-agent-test.yaml": error validating data: ValidationError(Clusteragent.spec): unknown field "nsToMonitorRegex" in com.appdynamics.v1alpha1.Clusteragent.spec; if you choose to ignore these errors, turn validation off with --validate=false

Re: Auto instrument not working

Morelz — Thu, 01 Jul 2021 14:53:11 GMT

Thanks Sattha

Can you just confirm a couple of things

1. Which version of K8S are you running?

2. What version of the Cluster Agent Operator are you running? If running version 1.14 or less you should have deployed the Operator with the below YAML File

cluster-agent-operator-1.14-or-less.yaml

You can try the below YAML and advise

apiVersion: appdynamics.com/v1alpha1
kind: Clusteragent
metadata:
  name: k8s-cluster-agent
  namespace: appdynamics
spec:
  appName: "SEACMACLUSTER-TH-DEVQA"
  controllerUrl: "https://xxxxxdev.saas.appdynamics.com"
  account: "xxxxxdev"
  image: "my-own-repository.org:5000/appdynamics/cluster-agent:latest"
  logLevel: Debug
  logFileSizeMb: 10
  logFileBackups: 5
  stdoutLogging: "true"
  serviceAccountName: appdynamics-cluster-agent
  nsToMonitor:
    - "appdynamics"
    - "cma-dev"
    - "cma-qa"
  proxyUrl: "http://172.30.60.46:80"
  proxyUser: "Th-xxxxx\\svc_localapp_dev"
  #
  # auto-instrumentation config
  #
  instrumentationMethod: Env
  nsToInstrument:
    - "appdynamics"
    - "cma-dev"
    - "cma-qa"
  appNameStrategy: manual
  defaultAppName: SEACMA-TH-NONPROD
  defaultEnv: JAVA_TOOL_OPTIONS
  instrumentationRules:
    - namespaceRegex: cma-dev
      language: java
      appName: SEACMA-TH-DEV
      containerMatchString: .*
      imageInfo:
        image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics

Re: Auto instrument not working

Pooja_Patel — Mon, 06 Mar 2023 19:57:43 GMT

did it work @sattha.puangput ? I am also facing the same issue with my containerized app.