Splunk AppDynamics
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Ask a Question

Auto instrument not working

sattha_puangput
Explorer
‎06-28-2021 09:38 AM

Hi All,

Currently, I try to implement cluster-operator and cluster-agent on K8s. the cluster metric shows under the "Server/Cluster" menu correctly. However, I would like to config auto-instrument for Java application but no hope follow the validate guide here is still not much helpful...

https://docs.appdynamics.com/21.5/en/infrastructure-visibility/monitor-kubernetes-with-the-cluster-a...

apiVersion: appdynamics.com/v1alpha1
kind: Clusteragent
metadata:
  name: k8s-cluster-agent
  namespace: appdynamics
spec:
  appName: "TEST-CLUSTER"
  controllerUrl: "https://xxx.appdynamics.com"
  account: "xxxdev"
  logLevel: DEBUG
  logFileSizeMb: 7
  logFileBackups: 6
  # docker image info
  image: "my.own.repository.org:5000/appdynamics/cluster-agent:latest"
  serviceAccountName: appdynamics-cluster-agent
  nsToMonitor:
    - "appdynamics"
    - "cma-dev"
    - "cma-qa"
  #
  # auto-instrumentation config
  #
  instrumentationMethod: Env
  nsToInstrumentRegex: "xxx-dev|xxx-qa"
  appNameStrategy: label
  defaultAppName: not-specific
  defaultCustomConfig: "-Dappdynamics.agent.maxMetrics=15000"
  defaultEnv: JAVA_TOOL_OPTIONS
  resourcesToInstrument:
    - Deployment
  imageInfo:
    java:
      image: "my.own.repository.org:5000/appdynamics/java-agent:latest"
      agentMountPath: /opt/appdynamics
  instrumentationRules:
    - namespaceRegex: "xxx-dev"
      language: java
      env: JAVA_TOOL_OPTIONS
      appNameLabel: app
      instrumentContainer: first
      imageInfo:
        image: "my.own.repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics
    - namespaceRegex: "xxx-qa"
      language: java
      env: JAVA_TOOL_OPTIONS
      appNameLabel: app
      instrumentContainer: first
      imageInfo:
        image: "my.own.repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics

But I got this....

[DEBUG]: 2021-06-28 16:12:36 - instrumentationconfig.go:628 - rule xxx-qa matches Deployment spring-api-productprofile in namespace xxx-qa with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-28 16:12:36 - instrumentationconfig.go:639 - Found a matching rule {xxx-qa  map[]   java first  -Dappdynamics.agent.maxMetrics=15000 JAVA_TOOL_OPTIONS map[agent-mount-path:/opt/appdynamics image:my-own-repository.org:5000/appdynamics/java-agent:latest] map[bci-enabled:true port:3892] 0 0 app  0 false []} for Deployment spring-api-productprofile in namespace xxx-qa with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-28 16:12:36 - instrumentationconfig.go:249 - Instrumentation state for Deployment spring-api-productprofile in namespace xxx-qa with labels map[app:spring-api-productprofile] is false
[DEBUG]: 2021-06-28 16:12:36 - instrumentationconfig.go:628 - rule xxx-dev matches Deployment spring-api-deliveries in namespace xxx-dev with labels map[app:spring-api-deliveries]
[DEBUG]: 2021-06-28 16:12:36 - instrumentationconfig.go:639 - Found a matching rule {xxx-dev  map[]   java first  -Dappdynamics.agent.maxMetrics=15000 JAVA_TOOL_OPTIONS map[agent-mount-path:/opt/appdynamics image:my-own-repository.org:5000/appdynamics/java-agent:latest] map[bci-enabled:true port:3892] 0 0 app  0 false []} for Deployment spring-api-deliveries in namespace xxx-dev with labels map[app:spring-api-deliveries]
[DEBUG]: 2021-06-28 16:12:36 - instrumentationconfig.go:249 - Instrumentation state for Deployment spring-api-deliveries in namespace xxx-dev with labels map[app:spring-api-deliveries] is false

Please advice

Labels (1)
Labels
Tags (1)
Reply

Pooja_Patel
New Member
‎03-06-2023 11:57 AM

did it work @sattha.puangput ? I am also facing the same issue with my containerized app.

0 Karma
Reply

Morelz
Builder
‎06-29-2021 06:05 AM

Hi Sattha

You have set the appNameStrategy : label

And in the instrumentation rules, you state that it should use the below label of the Java pods to name the Application.

Do you have a label in your java app's deployment spec that is named "app"? As it will look for that label to set the Application Name. If it's empty it wont register. Your logs show that the instrumentation rule is matching the pod so the rest seem to be correct.

appNameLabel: app

I would suggest to set the following, to just validate testing, before switching back to using labels to set the AppD Application Name

appNameStrategy: manual

Change 

appNameLabel: app to appName: XXX_DEV

                                             appName: XXX_QA

Then redeploy and let us know

Ciao

Reply

sattha_puangput
Explorer
‎06-29-2021 06:21 PM

Thank you for your advice, I update configure to this one now... but still got the same log output

apiVersion: appdynamics.com/v1alpha1
kind: Clusteragent
metadata:
  name: k8s-cluster-agent
  namespace: appdynamics
spec:
  appName: "SEACMACLUSTER-TH-DEVQA"
  controllerUrl: "https://xxx.saas.appdynamics.com"
  account: "xxx"
  logLevel: DEBUG
  logFileSizeMb: 7
  logFileBackups: 6
  # docker image info
  image: "my-own-repository.org:5000/appdynamics/cluster-agent:latest"
  serviceAccountName: appdynamics-cluster-agent
  nsToMonitor:
    - "appdynamics"
    - "cma-dev"
    - "cma-qa"
  proxyUrl: "http://xx.xx.xx.xx:80"
  proxyUser: "xxx\\svc_localapp_dev"
  #
  # auto-instrumentation config
  #
  instrumentationMethod: Env
  nsToInstrumentRegex: cma-dev|cma-qa
  appNameStrategy: manual
  defaultAppName: SEACMA-TH-NONPROD
  #defaultCustomConfig: "-Dappdynamics.agent.maxMetrics=15000"
  defaultEnv: JAVA_TOOL_OPTIONS
  resourcesToInstrument:
    - Deployment
  instrumentContainer: first
  imageInfo:
    java:
      image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
      agentMountPath: /opt/appdynamics
    dotnetcore:
      image: "docker.io/appdynamics/dotnet-core-agent:latest"
      agentMountPath: /opt/appdynamics
    nodejs:
      image: "docker.io/appdynamics/nodejs-agent:20.8.0-stretch-slimv10"
      agentMountPath: /opt/appdynamics
  instrumentationRules:
    - namespaceRegex: "cma-dev"
      appName: SEACMA-TH-DEV
      env: JAVA_TOOL_OPTIONS
      instrumentContainer: first
      imageInfo:
        image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics
    - namespaceRegex: "cma-qa"
      appName: SEACMA-TH-QA
      env: JAVA_TOOL_OPTIONS
      instrumentContainer: first
      imageInfo:
        image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics

Log

[DEBUG]: 2021-06-30 01:08:59 - instrumentationconfig.go:628 - rule cma-qa matches Deployment spring-api-productprofile in namespace cma-qa with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-30 01:08:59 - instrumentationconfig.go:639 - Found a matching rule {cma-qa  map[] SEACMA-TH-QA  java first   JAVA_TOOL_OPTIONS map[agent-mount-path:/opt/appdynamics image:my-own-repository:5000/appdynamics/java-agent:latest] map[bci-enabled:true port:3892] 0 0   0 false []} for Deployment spring-api-productprofile in namespace cma-qa with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-30 01:08:59 - instrumentationconfig.go:249 - Instrumentation state for Deployment spring-api-productprofile in namespace cma-qa with labels map[app:spring-api-productprofile] is false


[DEBUG]: 2021-06-30 01:08:59 - instrumentationconfig.go:628 - rule cma-dev matches Deployment spring-api-productprofile in namespace cma-dev with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-30 01:08:59 - instrumentationconfig.go:639 - Found a matching rule {cma-dev  map[] SEACMA-TH-DEV  java first   JAVA_TOOL_OPTIONS map[agent-mount-path:/opt/appdynamics image:my-own-repository.org:5000/appdynamics/java-agent:latest] map[bci-enabled:true port:3892] 0 0   0 false []} for Deployment spring-api-productprofile in namespace cma-dev with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-30 01:08:59 - instrumentationconfig.go:249 - Instrumentation state for Deployment spring-api-productprofile in namespace cma-dev with labels map[app:spring-api-productprofile] is false

What I try...

  1. check K8s deployment config to make sure that config is applied to the current "k8s-cluster-agent" pod
  2. delete and recreate application deployment but no init-container config apply (it should not I think since Instrument still false

Current dashboard screenshot...

image.pngclusterimage.pngapplication

0 Karma
Reply

Morelz
Builder
‎06-29-2021 11:58 PM

Thanks Sattha

Could you remove all instances of the following from the cluster agent yaml

instrumentContainer: first

and add the following to each of the 2 instrumentation rules

containerMatchString: .*

The redeploy, and check if it instruments the pods now

0 Karma
Reply

sattha_puangput
Explorer
‎06-30-2021 09:22 AM

I try to add the following config but it does not work...

instrumentContainer: select
containerMatchString: .*

apiVersion: appdynamics.com/v1alpha1
kind: Clusteragent
metadata:
  name: k8s-cluster-agent
  namespace: appdynamics
spec:
  appName: "SEACMACLUSTER-TH-DEVQA"
  controllerUrl: "https://xxxxxdev.saas.appdynamics.com"
  account: "xxxxxdev"
  logLevel: DEBUG
  logFileSizeMb: 7
  logFileBackups: 6
  # docker image info
  image: "my-own-repository.org:5000/appdynamics/cluster-agent:latest"
  serviceAccountName: appdynamics-cluster-agent
  ### Uncomment the following two lines if you need pull secrets
  #imagePullSecrets:
  #  name: "<your-docker-pull-secret-name>"
  nsToMonitor:
    - "appdynamics"
    - "cma-dev"
    - "cma-qa"
  proxyUrl: "http://172.30.60.46:80"
  proxyUser: "Th-xxxxx\\svc_localapp_dev"
  #
  # auto-instrumentation config
  #
  instrumentationMethod: Env
  nsToInstrumentRegex: cma-dev|cma-qa
  appNameStrategy: manual
  defaultAppName: SEACMA-TH-NONPROD
  instrumentContainer: select
  #defaultCustomConfig: "-Dappdynamics.agent.maxMetrics=15000"
  defaultEnv: JAVA_TOOL_OPTIONS
  resourcesToInstrument:
    - Deployment
  imageInfo:
    java:
      image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
      agentMountPath: /opt/appdynamics
    dotnetcore:
      image: "docker.io/appdynamics/dotnet-core-agent:latest"
      agentMountPath: /opt/appdynamics
    nodejs:
      image: "docker.io/appdynamics/nodejs-agent:20.8.0-stretch-slimv10"
      agentMountPath: /opt/appdynamics
  instrumentationRules:
    - namespaceRegex: "cma-dev"
      appName: SEACMA-TH-DEV
      env: JAVA_TOOL_OPTIONS
      instrumentContainer: select
      containerMatchString: .*
      imageInfo:
        image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics
    - namespaceRegex: "cma-qa"
      appName: SEACMA-TH-QA
      env: JAVA_TOOL_OPTIONS
      instrumentContainer: select
      containerMatchString: .*
      imageInfo:
        image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics

  Still same error

[DEBUG]: 2021-06-30 16:12:11 - instrumentationconfig.go:628 - rule cma-qa matches Deployment spring-api-productprofile in namespace cma-qa with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-30 16:12:11 - instrumentationconfig.go:639 - Found a matching rule {cma-qa  map[] SEACMA-TH-QA  java select .*  JAVA_TOOL_OPTIONS map[agent-mount-path:/opt/appdynamics image:pvnexusho001th.dmz.th-tesco.org:5000/appdynamics/java-agent:latest] map[bci-enabled:true port:3892] 0 0   0 false []} for Deployment spring-api-productprofile in namespace cma-qa with labels map[app:spring-api-productprofile]
[DEBUG]: 2021-06-30 16:12:11 - instrumentationconfig.go:249 - Instrumentation state for Deployment spring-api-productprofile in namespace cma-qa with labels map[app:spring-api-productprofile] is false

What I try

  • Update config according to your suggestion
  • Delete cluster-operator  K8S deployment
  • Delete cluster-agent K8S deployment
  • recreate both cluster-operator and cluster-agent
  • check deployment configuration change to what we need
  • delete application deployment and recreate it --- not thing happen no init-container
0 Karma
Reply

Morelz
Builder
‎07-01-2021 12:40 AM

Hi Sattha

Can you try the below YAML Config

apiVersion: appdynamics.com/v1alpha1
kind: Clusteragent
metadata:
  name: k8s-cluster-agent
  namespace: appdynamics
spec:
  appName: "SEACMACLUSTER-TH-DEVQA"
  controllerUrl: "https://xxxxxdev.saas.appdynamics.com"
  account: "xxxxxdev"
  image: "my-own-repository.org:5000/appdynamics/cluster-agent:latest"
  logLevel: Debug
  logFileSizeMb: 10
  logFileBackups: 5
  stdoutLogging: "true"
  serviceAccountName: appdynamics-cluster-agent
  nsToMonitorRegex: appdynamics|cma-dev|cma-qa
  proxyUrl: "http://172.30.60.46:80"
  proxyUser: "Th-xxxxx\\svc_localapp_dev"
  #
  # auto-instrumentation config
  #
  instrumentationMethod: Env
  nsToInstrumentRegex: cma-dev|cma-qa
  appNameStrategy: manual
  defaultAppName: SEACMA-TH-NONPROD
  defaultEnv: JAVA_TOOL_OPTIONS
  instrumentationRules:
    - namespaceRegex: cma-dev
      language: java
      appName: SEACMA-TH-DEV
      containerMatchString: .*
      imageInfo:
        image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics
Reply

sattha_puangput
Explorer
‎07-01-2021 07:40 AM

I try your .yaml file but when creating `cluster-agent.yaml` there is the following error

error: error validating "appdynamics/cluster-agent-test.yaml": error validating data: ValidationError(Clusteragent.spec): unknown field "nsToMonitorRegex" in com.appdynamics.v1alpha1.Clusteragent.spec; if you choose to ignore these errors, turn validation off with --validate=false
0 Karma
Reply

Morelz
Builder
‎07-01-2021 07:53 AM

Thanks Sattha

Can you just confirm a couple of things

1. Which version of K8S are you running?

2. What version of the Cluster Agent Operator are you running? If running version 1.14 or less you should have deployed the Operator with the below YAML File

cluster-agent-operator-1.14-or-less.yaml

You can try the below YAML and advise

apiVersion: appdynamics.com/v1alpha1
kind: Clusteragent
metadata:
  name: k8s-cluster-agent
  namespace: appdynamics
spec:
  appName: "SEACMACLUSTER-TH-DEVQA"
  controllerUrl: "https://xxxxxdev.saas.appdynamics.com"
  account: "xxxxxdev"
  image: "my-own-repository.org:5000/appdynamics/cluster-agent:latest"
  logLevel: Debug
  logFileSizeMb: 10
  logFileBackups: 5
  stdoutLogging: "true"
  serviceAccountName: appdynamics-cluster-agent
  nsToMonitor:
    - "appdynamics"
    - "cma-dev"
    - "cma-qa"
  proxyUrl: "http://172.30.60.46:80"
  proxyUser: "Th-xxxxx\\svc_localapp_dev"
  #
  # auto-instrumentation config
  #
  instrumentationMethod: Env
  nsToInstrument:
    - "appdynamics"
    - "cma-dev"
    - "cma-qa"
  appNameStrategy: manual
  defaultAppName: SEACMA-TH-NONPROD
  defaultEnv: JAVA_TOOL_OPTIONS
  instrumentationRules:
    - namespaceRegex: cma-dev
      language: java
      appName: SEACMA-TH-DEV
      containerMatchString: .*
      imageInfo:
        image: "my-own-repository.org:5000/appdynamics/java-agent:latest"
        agentMountPath: /opt/appdynamics
0 Karma
Reply
Get Updates on the Splunk Community!

Splunk Observability Synthetic Monitoring - Resolved Incident on Detector Alerts

We’ve discovered a bug that affected the auto-clear of Synthetic Detectors in the Splunk Synthetic Monitoring ...

Video | Tom’s Smartness Journey Continues

Remember Splunk Community member Tom Kopchak? If you caught the first episode of our Smartness interview ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud? Learn how unique features like ...