Splunk AppDynamics

Cluster Agent Pod Stuck in CrashLoopBackOff

sajo_sam
Explorer

I'm trying to deploy a cluster agent in my Kubernetes cluster to monitor the infrastructure using the kubectl CLI. I've followed the steps and executed these commands:

kubectl create -f cluster-agent-operator.yaml

kubectl -n appdynamics create secret generic cluster-agent-secret --from-literal=controller-key=<access-key>

kubectl create -f cluster-agent.yaml

However, the cluster agent pod is stuck in the "CrashLoopBackOff" state. The logs indicate an issue with the account access key:

[ERROR]: 2024-04-03 18:29:45 - main.go:183 - Account accessKey is not specified
[ERROR]: 2024-04-03 18:29:45 - main.go:184 - Please provide account accessKey before starting cluster-agent. Exiting...
  • I've verified that the cluster-agent-secret contains the controller-key with the correct access key value.

What could be causing this issue despite providing the access key in the secret? Are there any additional configuration steps I might be missing?

Reference :

https://docs.appdynamics.com/appd/22.x/latest/en/infrastructure-visibility/monitor-kubernetes-with-t...

Labels (1)
0 Karma

rganapavarapu
Path Finder

Hello @sajo.sam,

The error message you've encountered "agentregistrationmodule.go:132 - clusterId: -1" indicates that the cluster agent received incorrect controller information or that the controller is rejecting the registration.

To verify if you've configured the correct controller access key, you can execute the following command:

kubectl get secret cluster-agent-secret -n appdynamics -o jsonpath='{.data.controller-key}' | base64 --decode

Please note that DONOT share any controller sensitive information in this public platform.

 

Regarding the error mentioned in your comments:

[ERROR]: 2024-04-09 11:21:08 - agentregistrationmodule.go:131 - Failed to send agent registration request: Post "accountname.saas.appdynamics.com:8080/sim/v2/agent/clusterRegistration": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

This is likely due to a network connection issue. You can check the network connection from one of the running pods using the following command:

curl -v -k -u singularity-agent@<accountname>  https://<controllerhost>.saas.appdynamics.com:<port>/sim/v2/agent/clusterRegistration

 

Please check the above if that helpful.

Best Regards,
Rajesh Ganapavarapu

iamryan
Community Manager
Community Manager

Hi @Marcie.Sirbaugh,

I see you have an open ticket with the same error you asked Sajo about 

agentregistrationmodule.go:352

Perhaps you can continue to share any outcomes from that interaction with your ticket here with Sajo.

0 Karma

Marcie_Sirbaugh
Path Finder

I see this in your log, have you figured out why this is happening?
agentregistrationmodule.go:352 - "default" is not a valid namespace in your kubernetes cluster

0 Karma

sajo_sam
Explorer

Hy @Marcie.Sirbaugh 

I just looked into it but didn't get any information from this because the namespace default is already there but I don't know where i'm missing

0 Karma

iamryan
Community Manager
Community Manager

Hi @sajo.sam,

Did you get a chance to check out the TKB article or have you found a solution you can share?

0 Karma

sajo_sam
Explorer

Hi @Ryan.Paredez 


I tried but I'm stuck with another issue. The logs given below show it faces some errors with "Failed to send agent registration request: Post "accountname.saas.appdynamics.com:8080/sim/v2/agent/clusterRegistration ": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

[ERROR]: 2024-04-09 11:20:38 - secretconfig.go:68 - Problem With Getting /opt/appdynamics/cluster-agent/secret-volume/api-user Secret: open /opt/appdynamics/cluster-agent/secret-volume/api-user: no such file or directory

[INFO]: 2024-04-09 11:20:38 - main.go:78 - Kubernetes version: v1.29.0
[INFO]: 2024-04-09 11:20:38 - main.go:236 - Registering cluster agent with
 controller host : accountname.saas.appdynamics.com
 controller port : 8080
 account name : accountname
[WARNING]: 2024-04-09 11:20:38 - agentregistrationmodule.go:352 - "default" is not a valid namespace in your kubernetes cluster

[INFO]: 2024-04-09 11:20:38 - agentregistrationmodule.go:356 - Established connection to Kubernetes API
[INFO]: 2024-04-09 11:20:38 - agentregistrationmodule.go:68 - Cluster name: fromKube
[INFO]: 2024-04-09 11:20:38 - agentregistrationmodule.go:119 - Initial Agent registration
[ERROR]: 2024-04-09 11:21:08 - agentregistrationmodule.go:131 - Failed to send agent registration request: Post "accountname.saas.appdynamics.com:8080/sim/v2/agent/clusterRegistration": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[ERROR]: 2024-04-09 11:21:08 - agentregistrationmodule.go:132 - clusterId: -1
[ERROR]: 2024-04-09 11:21:08 - agentregistrationmodule.go:134 - Registration properties: {}
[INFO]: 2024-04-09 11:21:38 - agentregistrationmodule.go:119 - Initial Agent registration

^ Post edited by @Ryan.Paredez to remove mentions and links to Account name. For security and privacy reasons, please redact the name of your Account in Community posts.

0 Karma

iamryan
Community Manager
Community Manager
0 Karma
Get Updates on the Splunk Community!

Splunk AI Assistant for SPL | Key Use Cases to Unlock the Power of SPL

Splunk AI Assistant for SPL | Key Use Cases to Unlock the Power of SPL  The Splunk AI Assistant for SPL ...

Buttercup Games: Further Dashboarding Techniques (Part 5)

This series of blogs assumes you have already completed the Splunk Enterprise Search Tutorial as it uses the ...

Customers Increasingly Choose Splunk for Observability

For the second year in a row, Splunk was recognized as a Leader in the 2024 Gartner® Magic Quadrant™ for ...