Splunk AppDynamics

Error 502 after starting Java Agent

CommunityUser
Splunk Employee
Splunk Employee

Hello there,

We're deploying a app dynamics environment here to test some Java applications and the machines itself (hence, using java_agent and machine_agent).

However, as soon as we start the Java agent, we get connection to the controller, and it returns an error 502, as seen in the log bellow:

[AD Thread Pool-Global0] 14 Jul 2017 11:03:32,826 ERROR ConfigurationChannel - HTTP Request failed: HTTP/1.1 502 Bad Gateway
[AD Thread Pool-Global0] 14 Jul 2017 11:03:32,826  WARN ConfigurationChannel - Could not connect to the controller/invalid response from controller, cannot get initialization information, controller host [fanatics3c.saas.appdynamics.com], port[443], exception [null]
[AD Thread Pool-Global0] 14 Jul 2017 11:03:32,826 ERROR ConfigurationChannel - Exception: NULL
[AD Thread-Metric Reporter1] 14 Jul 2017 11:04:08,316  WARN MetricHandler - Metric Reporter Queue full. Dropping metrics.
[qtp568531308-75] 14 Jul 2017 11:04:30,100  INFO TransactionMonitor - Creating new BTContext for thread Id [75]
[AD Thread Pool-Global1] 14 Jul 2017 11:04:32,827  INFO ConfigurationChannel - Detected node meta info: [Name:ProcessID, Value:5554, Name:appdynamics.ip.addresses, Value:fe80:0:0:0:840:71ff:fe11:d5b2%eth0,10.151.16.50, Name:supportsDevMode, Value:true]
[AD Thread Pool-Global1] 14 Jul 2017 11:04:32,827  INFO ConfigurationChannel - Sending Registration request with: Application Name [3c-lpt], Tier Name [3c-lpt-promotion], Node Name [ip-10-151-16-50], Host Name [ip-10-151-16-50] Node Unique Local ID [ip-10-151-16-50], Version [Server Agent v4.3.3.2 GA #15225 r${env.GIT_COMMIT} 2813-4.3.3.next-analytics]
[AD Thread Pool-Global1] 14 Jul 2017 11:04:33,579 ERROR ConfigurationChannel - HTTP Request failed: HTTP/1.1 502 Bad Gateway
[AD Thread Pool-Global1] 14 Jul 2017 11:04:33,580  WARN ConfigurationChannel - Could not connect to the controller/invalid response from controller, cannot get initialization information, controller host [fanatics3c.saas.appdynamics.com], port[443], exception [null]
[AD Thread Pool-Global1] 14 Jul 2017 11:04:33,580 ERROR ConfigurationChannel - Exception: NULL
[qtp568531308-78] 14 Jul 2017 11:04:50,807  INFO TransactionMonitor - Creating new BTContext for thread Id [78]
[AD Thread-Metric Reporter0] 14 Jul 2017 11:05:08,316  WARN MetricHandler - Metric Reporter Queue full. Dropping metrics.
[AD Thread Pool-Global1] 14 Jul 2017 11:05:29,650  INFO ControllerTimeSkewHandler - Controller Time Skew Handler Run Aborted - Skew Check is Disabled
[AD Thread Pool-Global1] 14 Jul 2017 11:05:33,581  INFO ConfigurationChannel - Detected node meta info: [Name:ProcessID, Value:5554, Name:appdynamics.ip.addresses, Value:fe80:0:0:0:840:71ff:fe11:d5b2%eth0,10.151.16.50, Name:supportsDevMode, Value:true]
[AD Thread Pool-Global1] 14 Jul 2017 11:05:33,581  INFO ConfigurationChannel - Sending Registration request with: Application Name [3c-lpt], Tier Name [3c-lpt-promotion], Node Name [ip-10-151-16-50], Host Name [ip-10-151-16-50] Node Unique Local ID [ip-10-151-16-50], Version [Server Agent v4.3.3.2 GA #15225 r${env.GIT_COMMIT} 2813-4.3.3.next-analytics]
[AD Thread Pool-Global1] 14 Jul 2017 11:05:34,275 ERROR ConfigurationChannel - HTTP Request failed: HTTP/1.1 502 Bad Gateway
[AD Thread Pool-Global1] 14 Jul 2017 11:05:34,275  WARN ConfigurationChannel - Could not connect to the controller/invalid response from controller, cannot get initialization information, controller host [fanatics3c.saas.appdynamics.com], port[443], exception [null]
[AD Thread Pool-Global1] 14 Jul 2017 11:05:34,275 ERROR ConfigurationChannel - Exception: NULL
[qtp568531308-80] 14 Jul 2017 11:05:40,909  INFO TransactionMonitor - Creating new BTContext for thread Id [80]

Keep on mind that other agents (like Db and Machine agent) works well. We're not using the dbagent today, but we already tested it, and seems to be ok. The MachineAgent is reporting back right to the server and client, and show no errors.

Our controller-info.xml file is:

<controller-info>
    <controller-host>[REDACTED].saas.appdynamics.com</controller-host>
    <controller-port>443</controller-port>
    <controller-ssl-enabled>true</controller-ssl-enabled>

    <account-name>[REDACTED]</account-name>
    <account-access-key>[REDACTED]</account-access-key>

    <application-name>3c-lpt</application-name>
    <tier-name>3c-lpt-promotion</tier-name>
    <node-name>ip-10-151-16-50</node-name>
</controller-info>
Labels (1)
0 Karma

Rajesh_Putta
Communicator

Hi,

Could you please send us zipped agent logs to validate few things.

Thanks

Rajesh

0 Karma

CommunityUser
Splunk Employee
Splunk Employee

Good morning Rajesh,

I found the solution last Friday. our issue here was provoked due to a new behavior of the Java Agent, which uses a versioning system since the very first 4.3.x agent. For some reason, the local default configuration hadn't some parameters comented out (like SSL connection to controller) conflicting with our global controller-info.xml file. 

The solution was just replace the global config file for an additional copy of our generated controller-info.xml

0 Karma

Rajesh_Putta
Communicator

Glad to hear this good news. Please let us know if we can solve this one,

Thanks

Rajesh

0 Karma
Get Updates on the Splunk Community!

Security Professional: Sharpen Your Defenses with These .conf25 Sessions

Sooooooooooo, guess what. .conf25 is almost here, and if you're on the Security Learning Path, this is your ...

First Steps with Splunk SOAR

Our first step was to gather a list of the playbooks we wanted and to sort them by priority.  Once this list ...

How To Build a Self-Service Observability Practice with Splunk Observability Cloud

If you’ve read our previous post on self-service observability, you already know what it is and why it ...