AppD Archive
×

Are you a member of the Splunk Community?

Sign in or Register with your Splunk account to get your questions answered, access valuable resources and connect with experts!
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Ask a Question

Unable to get AppAgentConfigurationResponse from controller

CommunityUser
Splunk Employee
Splunk Employee
‎09-18-2015 11:53 AM

I'm unable to monitor my java application (running on Sun/Oracle jdk 1.8.0_60). I'm using the latest java agent 4.1.3.1 and my controller is at the latest version 4.1.3.1. And my machine agent, also 4.1.3.1, is running fine and is pushing data to my controller. Both the machine and the app agent are using the exact same settings so why is the app agent not able to push any data to my controller? I tried downgrading the app agent and I tried various combinaison of values inside my controller-info.xml file but still no luck.

The machine agent is using java build 1.7.0_79-b15 and my app agent is using build 1.8.0_60-b27. So could it be an issue with java 8? Below are some logs from my app agent and from my controller. As you can see the agent is complaining "name is not specified" and on the controller we can see "Machine  Id: 0  Node Id: 0  name is not specified". This is weird.

APP AGENT LOG

[AD Thread Pool-Global0] 18 Sep 2015 14:08:32,254  INFO ConfigurationChannel - Sending Registration request with: Application Name [x], Tier Name [y], Node Name [z], Host Name [z@z.com] Node Unique Local ID [z], Version [Server Agent v4.1.3.1 GA #9271 r9a9db05086a674717c7d051efe94be349a78de23 19-4.1.3.next-build]
[AD Thread Pool-Global0] 18 Sep 2015 14:08:32,296  WARN ConfigurationChannel - ResponseReadException creating Response Wrapper [xb], : com.singularity.ee.rest.c: Error in controller in processing binary request AppAgent Config Data  - name is not specified

CONTROLLER LOG

[#|2015-09-18T14:26:33.177-0400|SEVERE|glassfish3.1.2|com.singularity.ee.controller.beans.ExceptionHandlingInterceptor|_ThreadID=58;_ThreadName=Thread-5;|Encountered a server exception
com.singularity.ee.controller.api.exceptions.IllegalObjectStateException: name is not specified
        at com.singularity.ee.controller.entity.AEntity.fromDTO(AEntity.java:346)
        at com.singularity.ee.controller.entity.RuntimeEnvironmentPropertyEntity.setPropertyValues(RuntimeEnvironmentPropertyEntity.java:143)
        at com.singularity.ee.controller.entity.RuntimeEnvironmentPropertyEntity.doFromDTO(RuntimeEnvironmentPropertyEntity.java:176)
        at com.singularity.ee.controller.entity.RuntimeEnvironmentPropertyEntity.doFromDTO(RuntimeEnvironmentPropertyEntity.java:40)
        at com.singularity.ee.controller.entity.AEntity.fromDTO(AEntity.java:344)
        at com.singularity.ee.controller.entity.AgentEntity.addToHistoryIfChanged(AgentEntity.java:272)
        at com.singularity.ee.controller.entity.AgentEntity.addVmSystemPropertiesToHistoryIfChanged(AgentEntity.java:368)
        at sun.reflect.GeneratedMethodAccessor665.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:197)
        at com.singularity.ee.controller.entity.AgentEntity_$$_javassist_195.addVmSystemPropertiesToHistoryIfChanged(AgentEntity_$$_javassist_195.java)
        at com.singularity.ee.controller.beans.model.ApplicationComponentManagerBean.updateApplicationComponentNodeAgentRuntimeState(ApplicationComponentManagerBean.java:2753)
        at sun.reflect.GeneratedMethodAccessor663.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1052)
        at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1124)
        at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:5388)
        at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:619)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571)
        at com.appdynamics.platform.KRolesInterceptor.methodInterceptor(KRolesInterceptor.java:47)
        at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571)
        at com.singularity.ee.controller.beans.EntityManagerInterceptor.manageEntityManagerLifecycle(EntityManagerInterceptor.java:53)
        at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571)
        at com.singularity.ee.controller.beans.ExceptionHandlingInterceptor.handleExceptions(ExceptionHandlingInterceptor.java:60)
        at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571)

        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doAround(SystemInterceptorProxy.java:162)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:144)
        at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
        at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:370)
        at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:5360)
        at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:5348)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:214)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:89)
        at com.sun.proxy.$Proxy503.updateApplicationComponentNodeAgentRuntimeState(Unknown Source)
        at com.singularity.ee.controller.beans.model.ApplicationComponentManagerBean$4.doEntityUpdate(ApplicationComponentManagerBean.java:2631)
        at com.singularity.ee.controller.beans.model.ApplicationComponentManagerBean$4.doEntityUpdate(ApplicationComponentManagerBean.java:2627)
        at com.singularity.ee.controller.beans.AControllerBean.doEntityUpdateOperationWithOptimisticLockRecovery(AControllerBean.java:238)
        at com.singularity.ee.controller.beans.model.ApplicationComponentManagerBean.updateApplicationComponentNodeAgentRuntimeStateOptimisticLockingRecovery(ApplicationComponentManagerBean.java:2625)
        at sun.reflect.GeneratedMethodAccessor662.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1052)
        at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1124)
        at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:5388)
        at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:619)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571)
        at com.appdynamics.platform.KRolesInterceptor.methodInterceptor(KRolesInterceptor.java:47)
        at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571)
        at com.singularity.ee.controller.beans.EntityManagerInterceptor.manageEntityManagerLifecycle(EntityManagerInterceptor.java:53)
        at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571)
        at com.singularity.ee.controller.beans.ExceptionHandlingInterceptor.handleExceptions(ExceptionHandlingInterceptor.java:60)
        at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doAround(SystemInterceptorProxy.java:162)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:144)
        at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
        at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:370)
        at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:5360)
        at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:5348)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:214)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:89)

        at com.sun.proxy.$Proxy503.updateApplicationComponentNodeAgentRuntimeStateOptimisticLockingRecovery(Unknown Source)
        at com.singularity.ee.controller.servlet.handlers.q.updateApplicationComponentNodeAgentRuntimeState(q.java:378)
        at com.singularity.ee.controller.servlet.handlers.q.handleAgentRegistrationRequest(q.java:280)
        at com.singularity.ee.controller.servlet.handlers.q._handleRequest(q.java:211)
        at com.singularity.ee.controller.servlet.handlers.a.handleRequest(a.java:451)
        at com.singularity.ee.controller.servlet.ControllerAgentServlet.doPost(ControllerAgentServlet.java:114)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:688)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
        at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1550)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:343)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:217)
        at com.singularity.ee.controller.servlet.CsrfFilter.doFilter(CsrfFilter.java:84)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:217)
        at com.singularity.ee.controller.servlet.RestApiTypeMarkingFilter.doFilter(RestApiTypeMarkingFilter.java:46)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:217)
        at com.singularity.ee.controller.servlet.RequestOriginMarkingFilter.doFilter(RequestOriginMarkingFilter.java:26)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:217)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:217)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:279)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
        at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
        at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:317)
        at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
        at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:860)
        at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:757)
        at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1056)
        at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:229)
        at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
        at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
        at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
        at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
        at com.sun.grizzly.ContextTask.run(ContextTask.java:121)
        at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:554)
        at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:533)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalArgumentException: name is not specified
        at com.singularity.ee.controller.entity.NameValuePairEntity.setName(NameValuePairEntity.java:60)
        at com.singularity.ee.controller.entity.NameValuePairEntity.doFromDTO(NameValuePairEntity.java:76)
        at com.singularity.ee.controller.entity.NameValuePairEntity.doFromDTO(NameValuePairEntity.java:20)
        at com.singularity.ee.controller.entity.AEntity.fromDTO(AEntity.java:344)
        ... 149 more
|#]

[#|2015-09-18T14:26:33.178-0400|SEVERE|glassfish3.1.2|com.appdynamics.AGENT.AGENT_CONFIGURATION|_ThreadID=58;_ThreadName=Thread-5;|Failed handling transaction config request.  Machine  Id: 0  Node Id: 0  name is not specified|#]

0 Karma
Reply

Arun_Dasetty
Super Champion
‎09-21-2015 06:10 AM

Hi,

- Can you please collect archive of <AppServerAgent_install_dir>/logs folder and attach to this post in addition to logs from machine agent directory and server.log from <Controller_dir>/logs if this is onPremise UI

- While you provide above logs , Can you please provide app,tier-name and node-name in <agent_dir>/conf/contorller-info.xml manually and see how it goes after jvm restart?

0 Karma
Reply

CommunityUser
Splunk Employee
Splunk Employee
‎09-21-2015 07:32 AM

Here's a zip file that contains my agent and controller logs. And I tried forcing the various controller-info.xml variables directly on the java command line (ie: -Dappdynamics.controller.hostName=x -Dappdynamics.controller.port=x -Dappdynamics.agent.applicationName=x -Dappdynamics.agent.tierName=x -Dappdynamics.agent.nodeName=x) but it's still not working.

AppAgentConfigurationResponse.zip
0 Karma
Reply

Arun_Dasetty
Super Champion
‎09-22-2015 06:17 AM

Hi

- Can you help to try below steps:


a) define one simple java class say Test.java with main and some thread.sleep in main method

b) now run the agent as follows:
shell> java -Dappdynamics.controller.hostName=appdyn-02.touchtunes.com -Dappdynamics.controller.port=8090 -Dappdynamics.agent.uniqueHostId=g3jab-1800-01_1 -Dappdynamics.agent.applicationName=jukeserver1 -Dappdynamics.agent.tierName=prod1 -Dappdynamics.agent.nodeName=g3jab-1800-01_1 -javaagent:/opt/appdynamics/AppServerAgent/ver4.1.3.1/javaagent.jar Test

And see whether agent registered to new app in UI. We see below errors in logs and we understood that you have changes account access key while uploading logs:

[Thread-0] 21 Sep 2015 10:18:29,001 INFO XMLConfigManager - XML Agent Account Info Resolver using account access

key [XXX]

[#|2015-09-21T10:19:36.328-0400|SEVERE|glassfish3.1.2|com.appdynamics.AGENT.AGENT_CONFIGURATION|_ThreadID=311;_ThreadName=Thread-5;|Failed handling transaction config request. Machine Id: 0 Node Id: 0 name is not specified|#]

[#|2015-09-21T10:19:42.363-0400|SEVERE|glassfish3.1.2|com.singularity.ee.controller.beans.agent.db.PerfDataUtils|_ThreadID=20403;_ThreadName=Thread-5;|sql query execute on shard failed:
java.sql.SQLException: Failed to read from the .par file


- Can you attach below files if issue persits/opt/appdynamics/AppServerAgent/ver4

a) /opt/appdynamics/AppServerAgent/ver4.1.3.1/conf/controller-info.xml and /opt/appdynamics/AppServerAgent/conf/controller-info.xml

b) latest agent logs and server.logs

0 Karma
Reply

CommunityUser
Splunk Employee
Splunk Employee
‎09-22-2015 01:13 PM

I restarted my application with java 7 (jdk1.7.0_75) instead of java 8 (jdk1.8.0_60) and the agent is now pushing data to my controller. But unfortunately I cannot run my application on java 7 so keeping java 7 is not an option. So it look like there is really something wrong with the agent (4.1.3.1 in this case) and/or my application and/or java 8.

Also I don't have access to the code and I'm not a programmer so adding some testing class to my application is not really an option. And I also tested the 4.1.1.1 agent and I'm having the same issue. Maybe downgrading to a 3.x agent could fix the problem?

0 Karma
Reply

CommunityUser
Splunk Employee
Splunk Employee
‎09-22-2015 03:24 PM

I downgraded the agent to version 3.9.3.0 and it's now talking to the controller. I will make some tests to make sure that transactions are being seen inside the controller.

0 Karma
Reply

Arun_Dasetty
Super Champion
‎09-23-2015 06:46 AM

Hi,

We do not expect such behavior due to differnet jdk as it is expected to support by 4.x agent, sure keep us posted how it goes with BT detection, if you do not see any BTs, you could define Custom pojo rules https://docs.appdynamics.com/display/PRO41/POJO+Entry+Points in UI to see if that helps

Simple program when we said looks like  below:

http://www.javatpoint.com/simple-program-of-java

0 Karma
Reply
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Take the Survey
Get Updates on the Splunk Community!

Introducing Splunk 10.0: Smarter, Faster, and More Powerful Than Ever

Now On Demand Whether you're managing complex deployments or looking to future-proof your data ...

Community Content Calendar, September edition

Welcome to another insightful post from our Community Content Calendar! We're thrilled to continue bringing ...

Splunkbase Unveils New App Listing Management Public Preview

Splunkbase Unveils New App Listing Management Public PreviewWe're thrilled to announce the public preview of ...