AppDynamics controller exception in logfile

CommunityUser — Tue, 12 May 2015 10:58:09 GMT

Hi,

my server.log file is full of this error:

[#|2015-05-12T12:53:48.449+0200|SEVERE|glassfish3.1.2|com.appdynamics.AGENT|_ThreadID=58;_ThreadName=Thread-5;|Failed handling Write Query Stats collection request.
javax.ejb.EJBException
	at com.sun.ejb.containers.BaseContainer.processSystemException(BaseContainer.java:5215)
	at com.sun.ejb.containers.BaseContainer.completeNewTx(BaseContainer.java:5113)
	at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:4901)
	at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2045)
	at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:1994)
	at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:222)
	at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:89)
	at com.sun.proxy.$Proxy456.write(Unknown Source)
	at com.singularity.ee.controller.servlet.handlers.dbmon.WriteQueryStatRequestHandler._handleRequest(WriteQueryStatRequestHandler.java:43)
	at com.singularity.ee.controller.servlet.handlers.a.handleRequest(a.java:411)
	at com.singularity.ee.controller.servlet.ControllerAgentServlet.doPost(ControllerAgentServlet.java:114)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:688)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
	at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1550)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
	at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
	at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
	at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
	at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:317)
	at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
	at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:860)
	at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:757)
	at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1056)
	at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:229)
	at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
	at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
	at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
	at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
	at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
	at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
	at com.sun.grizzly.ContextTask.run(ContextTask.java:121)
	at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:554)
	at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:533)
	at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: An unrecoverable server error occurred. Please see server log for details.[STACK TRACE: com.appdynamics.analytics.shared.rest.exceptions.ClientException: Could not execute request to http://localhost:9080/v1/events/dbmon-wait-time
	at com.appdynamics.analytics.shared.rest.client.utils.GenericHttpRequestBuilder.getResponse(GenericHttpRequestBuilder.java:202)
	at com.appdynamics.analytics.shared.rest.client.utils.GenericHttpRequestBuilder.executeAndReturnRawResponseString(GenericHttpRequestBuilder.java:216)
	at com.appdynamics.analytics.shared.rest.client.eventservice.DefaultEventServiceClient.registerEventType(DefaultEventServiceClient.java:133)
	at sun.reflect.GeneratedMethodAccessor1177.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at com.singularity.ee.controller.beans.analytics.client.AccountCreatingAnalyticsClient$ProxyingEventServiceClient.invoke(AccountCreatingAnalyticsClient.java:98)
	at com.sun.proxy.$Proxy613.registerEventType(Unknown Source)
	at com.appdynamics.analytics.shared.rest.client.DefaultAnalyticsClient.registerEventType(DefaultAnalyticsClient.java:30)
	at com.singularity.ee.controller.beans.dbmon.DBMonElasticsearchManagerBean.registerEventTypeIfAbsent(DBMonElasticsearchManagerBean.java:144)
	at com.singularity.ee.controller.beans.dbmon.DBMonElasticsearchManagerBean.registerAccountIfAbsent(DBMonElasticsearchManagerBean.java:116)
	at com.singularity.ee.controller.beans.dbmon.DBMonElasticsearchManagerBean.registerAccount(DBMonElasticsearchManagerBean.java:98)
	at com.singularity.ee.controller.beans.dbmon.DBMonElasticsearchManagerBean.getEventServiceClient(DBMonElasticsearchManagerBean.java:83)
	at com.singularity.ee.controller.beans.dbmon.DBMonElasticsearchManagerBean.publishEvents(DBMonElasticsearchManagerBean.java:157)
	at com.singularity.ee.controller.beans.dbmon.DBMonElasticsearchManagerBean.writeMeasurement(DBMonElasticsearchManagerBean.java:208)
	at com.singularity.ee.controller.beans.dbmon.DBMonElasticsearchManagerBean.write(DBMonElasticsearchManagerBean.java:186)
	at sun.reflect.GeneratedMethodAccessor1530.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1052)
	at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1124)
	at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:5388)
	at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:619)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
	at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571)
	at com.singularity.ee.controller.beans.EntityManagerInterceptor.manageEntityManagerLifecycle(EntityManagerInterceptor.java:50)
	at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
	at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571)
	at com.singularity.ee.controller.beans.ExceptionHandlingInterceptor.handleExceptions(ExceptionHandlingInterceptor.java:60)
	at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
	at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571)
	at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doAround(SystemInterceptorProxy.java:162)
	at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:144)
	at sun.reflect.GeneratedMethodAccessor88.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
	at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:370)
	at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:5360)
	at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:5348)
	at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:214)
	at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:89)
	at com.sun.proxy.$Proxy456.write(Unknown Source)
	at com.singularity.ee.controller.servlet.handlers.dbmon.WriteQueryStatRequestHandler._handleRequest(WriteQueryStatRequestHandler.java:43)
	at com.singularity.ee.controller.servlet.handlers.a.handleRequest(a.java:411)
	at com.singularity.ee.controller.servlet.ControllerAgentServlet.doPost(ControllerAgentServlet.java:114)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:688)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
	at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1550)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
	at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
	at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
	at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
	at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:317)
	at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
	at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:860)
	at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:757)
	at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1056)
	at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:229)
	at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
	at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
	at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
	at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
	at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
	at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
	at com.sun.grizzly.ContextTask.run(ContextTask.java:121)
	at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:554)
	at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:533)
	at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9080 [localhost/127.0.0.1] failed: Connection refused
	at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:318)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
	at com.appdynamics.analytics.shared.rest.client.utils.GenericHttpRequestBuilder.getResponse(GenericHttpRequestBuilder.java:200)
	... 80 more
Caused by: java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
	at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
	at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
	at java.net.SocksSocketImpl.connect(Unknown Source)
	at java.net.Socket.connect(Unknown Source)
	at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)
	at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:123)
	... 90 more
]
	at com.singularity.ee.controller.beans.ExceptionHandlingInterceptor.handleExceptions(ExceptionHandlingInterceptor.java:101)
	at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
	at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571)
	at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doAround(SystemInterceptorProxy.java:162)
	at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:144)
	at sun.reflect.GeneratedMethodAccessor88.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861)
	at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
	at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:370)
	at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:5360)
	at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:5348)
	at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:214)
	... 31 more
|#]

Do you have an idea about this problem?

Regards,

Thomas

Re: AppDynamics controller exception in logfile

Arun_Dasetty — Tue, 12 May 2015 14:06:06 GMT

Hi Thomas,

It looks one of the appserver agent is trying to send data to Analytics events processor and by default the events processor that comes with Controller installation does not start automatically and we need to start manually using below command:

shell> cd <Controller_install_dir>/bin

shell> controller.bat/sh start-events-service

refer doc https://docs.appdynamics.com/display/PRO40/Start+or+Stop+the+Controller#StartorStoptheController-StartandStoptheEventsService

The errors you see in controller should not affect any controller performance apart from failing to process transaction/log analytics data sent by agent, Hope that information helps.

Regards,

Arun

Re: AppDynamics controller exception in logfile

CommunityUser — Wed, 13 May 2015 12:14:36 GMT

I do not want to use Analytics.

How can I find out which agents tries to send data to Analytics?

I have more that 100 agents and without any informations it is hard to find to correct agent.

Regards,

Thomas

Re: AppDynamics controller exception in logfile

Arun_Dasetty — Wed, 13 May 2015 13:54:36 GMT

Hi Thomas,

We might need to check step wise , can you check which applications when selected from app list in screen , Analytics -> Configuration at URI controller/#/location=ANALYTICS_HOME&timeRange=last_15_minutes.BEFORE_NOW.-1.-1.15 are having Analytics checkbox "enable analytics data collection for <app name>" is selected that will tell which application for which analytics enabled and in further screens in same section you should see which BTs from which tiers have this feature enabled

If no application in analytics config screen are enabled for anaytics for any app, it means the error is raised by controller internal agent or the error could be ignored given that controller tries to connect to events service by default expecting to be up.

Hope that information helps.

Regards,

Arun

Re: AppDynamics controller exception in logfile

CommunityUser — Wed, 13 May 2015 14:42:49 GMT

Hi Arun,

no application is enabled for analytics!

But is look that each BT (names are obfuscated) reports analytic data&colon;

Could that be a reason for the behaviour of my controller?Should I remove all BTs?

Regards,

Thomas

Re: AppDynamics controller exception in logfile

Arun_Dasetty — Thu, 14 May 2015 14:30:37 GMT

Hi Thomas,

Yes we request you to try the changes and if error still raises start-events-service using controller.sh/bat file and see how it goes, Keep us posted how it goes.

Regards,

Arun

topic Re: AppDynamics controller exception in logfile in AppD Archive

AppDynamics controller exception in logfile

Re: AppDynamics controller exception in logfile

Re: AppDynamics controller exception in logfile

Re: AppDynamics controller exception in logfile

Re: AppDynamics controller exception in logfile

Re: AppDynamics controller exception in logfile