topic Re: Cluster Agent Pod Stuck in CrashLoopBackOff in Splunk AppDynamics

Cluster Agent Pod Stuck in CrashLoopBackOff

sajo_sam — Thu, 04 Apr 2024 16:40:16 GMT

I'm trying to deploy a cluster agent in my Kubernetes cluster to monitor the infrastructure using the kubectl CLI. I've followed the steps and executed these commands:

kubectl create -f cluster-agent-operator.yaml

kubectl -n appdynamics create secret generic cluster-agent-secret --from-literal=controller-key=<access-key>

kubectl create -f cluster-agent.yaml

However, the cluster agent pod is stuck in the "CrashLoopBackOff" state. The logs indicate an issue with the account access key:

[ERROR]: 2024-04-03 18:29:45 - main.go:183 - Account accessKey is not specified
[ERROR]: 2024-04-03 18:29:45 - main.go:184 - Please provide account accessKey before starting cluster-agent. Exiting...

I've verified that the cluster-agent-secret contains the controller-key with the correct access key value.

What could be causing this issue despite providing the access key in the secret? Are there any additional configuration steps I might be missing?

Reference :

https://docs.appdynamics.com/appd/22.x/latest/en/infrastructure-visibility/monitor-kubernetes-with-the-cluster-agent/install-the-cluster-agent/install-the-cluster-agent-with-the-kubernetes-cli

Re: Cluster Agent Pod Stuck in CrashLoopBackOff

iamryan — Fri, 05 Apr 2024 16:45:57 GMT

Hi @sajo.sam,

I found this TKB article. Please check it out and see if it helps.
https://community.appdynamics.com/t5/Knowledge-Base/How-do-I-debug-common-Linux-Private-Synthetic-Agent-issues/ta-p/51547

Re: Cluster Agent Pod Stuck in CrashLoopBackOff

iamryan — Tue, 09 Apr 2024 15:53:03 GMT

Hi @sajo.sam,

Did you get a chance to check out the TKB article or have you found a solution you can share?

Re: Cluster Agent Pod Stuck in CrashLoopBackOff

sajo_sam — Tue, 09 Apr 2024 16:05:21 GMT

Hi @Ryan.Paredez

I tried but I'm stuck with another issue. The logs given below show it faces some errors with "Failed to send agent registration request: Post "accountname.saas.appdynamics.com:8080/sim/v2/agent/clusterRegistration ": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

[ERROR]: 2024-04-09 11:20:38 - secretconfig.go:68 - Problem With Getting /opt/appdynamics/cluster-agent/secret-volume/api-user Secret: open /opt/appdynamics/cluster-agent/secret-volume/api-user: no such file or directory

[INFO]: 2024-04-09 11:20:38 - main.go:78 - Kubernetes version: v1.29.0
[INFO]: 2024-04-09 11:20:38 - main.go:236 - Registering cluster agent with
 controller host : accountname.saas.appdynamics.com
 controller port : 8080
 account name : accountname
[WARNING]: 2024-04-09 11:20:38 - agentregistrationmodule.go:352 - "default" is not a valid namespace in your kubernetes cluster

[INFO]: 2024-04-09 11:20:38 - agentregistrationmodule.go:356 - Established connection to Kubernetes API
[INFO]: 2024-04-09 11:20:38 - agentregistrationmodule.go:68 - Cluster name: fromKube
[INFO]: 2024-04-09 11:20:38 - agentregistrationmodule.go:119 - Initial Agent registration
[ERROR]: 2024-04-09 11:21:08 - agentregistrationmodule.go:131 - Failed to send agent registration request: Post "accountname.saas.appdynamics.com:8080/sim/v2/agent/clusterRegistration": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[ERROR]: 2024-04-09 11:21:08 - agentregistrationmodule.go:132 - clusterId: -1
[ERROR]: 2024-04-09 11:21:08 - agentregistrationmodule.go:134 - Registration properties: {}
[INFO]: 2024-04-09 11:21:38 - agentregistrationmodule.go:119 - Initial Agent registration

^ Post edited by @Ryan.Paredez to remove mentions and links to Account name. For security and privacy reasons, please redact the name of your Account in Community posts.

Re: Cluster Agent Pod Stuck in CrashLoopBackOff

Marcie_Sirbaugh — Wed, 10 Apr 2024 13:39:09 GMT

I see this in your log, have you figured out why this is happening?
agentregistrationmodule.go:352 - "default" is not a valid namespace in your kubernetes cluster

Re: Cluster Agent Pod Stuck in CrashLoopBackOff

sajo_sam — Wed, 10 Apr 2024 16:20:08 GMT

Hy @Marcie.Sirbaugh

I just looked into it but didn't get any information from this because the namespace default is already there but I don't know where i'm missing

Re: Cluster Agent Pod Stuck in CrashLoopBackOff

iamryan — Thu, 11 Apr 2024 15:46:37 GMT

Hi @Marcie.Sirbaugh,

I see you have an open ticket with the same error you asked Sajo about

agentregistrationmodule.go:352

Perhaps you can continue to share any outcomes from that interaction with your ticket here with Sajo.

Re: Cluster Agent Pod Stuck in CrashLoopBackOff

rganapavarapu — Tue, 14 May 2024 20:48:42 GMT

Hello @sajo.sam,

The error message you've encountered "agentregistrationmodule.go:132 - clusterId: -1" indicates that the cluster agent received incorrect controller information or that the controller is rejecting the registration.

To verify if you've configured the correct controller access key, you can execute the following command:

kubectl get secret cluster-agent-secret -n appdynamics -o jsonpath='{.data.controller-key}' | base64 --decode

Please note that DONOT share any controller sensitive information in this public platform.

Regarding the error mentioned in your comments:

[ERROR]: 2024-04-09 11:21:08 - agentregistrationmodule.go:131 - Failed to send agent registration request: Post "accountname.saas.appdynamics.com:8080/sim/v2/agent/clusterRegistration": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

This is likely due to a network connection issue. You can check the network connection from one of the running pods using the following command:

curl -v -k -u singularity-agent@<accountname> https://<controllerhost>.saas.appdynamics.com:<port>/sim/v2/agent/clusterRegistration

Please check the above if that helpful.

Best Regards,
Rajesh Ganapavarapu