Hi,
my server.log file is full of this error:
[#|2015-05-12T12:53:48.449+0200|SEVERE|glassfish3.1.2|com.appdynamics.AGENT|_ThreadID=58;_ThreadName=Thread-5;|Failed handling Write Query Stats collection request. javax.ejb.EJBException at com.sun.ejb.containers.BaseContainer.processSystemException(BaseContainer.java:5215) at com.sun.ejb.containers.BaseContainer.completeNewTx(BaseContainer.java:5113) at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:4901) at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2045) at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:1994) at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:222) at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:89) at com.sun.proxy.$Proxy456.write(Unknown Source) at com.singularity.ee.controller.servlet.handlers.dbmon.WriteQueryStatRequestHandler._handleRequest(WriteQueryStatRequestHandler.java:43) at com.singularity.ee.controller.servlet.handlers.a.handleRequest(a.java:411) at com.singularity.ee.controller.servlet.ControllerAgentServlet.doPost(ControllerAgentServlet.java:114) at javax.servlet.http.HttpServlet.service(HttpServlet.java:688) at javax.servlet.http.HttpServlet.service(HttpServlet.java:770) at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1550) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161) at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231) at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:317) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195) at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:860) at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:757) at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1056) at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:229) at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137) at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104) at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90) at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79) at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54) at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59) at com.sun.grizzly.ContextTask.run(ContextTask.java:121) at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:554) at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:533) at java.lang.Thread.run(Unknown Source) Caused by: java.lang.RuntimeException: An unrecoverable server error occurred. Please see server log for details.[STACK TRACE: com.appdynamics.analytics.shared.rest.exceptions.ClientException: Could not execute request to http://localhost:9080/v1/events/dbmon-wait-time at com.appdynamics.analytics.shared.rest.client.utils.GenericHttpRequestBuilder.getResponse(GenericHttpRequestBuilder.java:202) at com.appdynamics.analytics.shared.rest.client.utils.GenericHttpRequestBuilder.executeAndReturnRawResponseString(GenericHttpRequestBuilder.java:216) at com.appdynamics.analytics.shared.rest.client.eventservice.DefaultEventServiceClient.registerEventType(DefaultEventServiceClient.java:133) at sun.reflect.GeneratedMethodAccessor1177.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.singularity.ee.controller.beans.analytics.client.AccountCreatingAnalyticsClient$ProxyingEventServiceClient.invoke(AccountCreatingAnalyticsClient.java:98) at com.sun.proxy.$Proxy613.registerEventType(Unknown Source) at com.appdynamics.analytics.shared.rest.client.DefaultAnalyticsClient.registerEventType(DefaultAnalyticsClient.java:30) at com.singularity.ee.controller.beans.dbmon.DBMonElasticsearchManagerBean.registerEventTypeIfAbsent(DBMonElasticsearchManagerBean.java:144) at com.singularity.ee.controller.beans.dbmon.DBMonElasticsearchManagerBean.registerAccountIfAbsent(DBMonElasticsearchManagerBean.java:116) at com.singularity.ee.controller.beans.dbmon.DBMonElasticsearchManagerBean.registerAccount(DBMonElasticsearchManagerBean.java:98) at com.singularity.ee.controller.beans.dbmon.DBMonElasticsearchManagerBean.getEventServiceClient(DBMonElasticsearchManagerBean.java:83) at com.singularity.ee.controller.beans.dbmon.DBMonElasticsearchManagerBean.publishEvents(DBMonElasticsearchManagerBean.java:157) at com.singularity.ee.controller.beans.dbmon.DBMonElasticsearchManagerBean.writeMeasurement(DBMonElasticsearchManagerBean.java:208) at com.singularity.ee.controller.beans.dbmon.DBMonElasticsearchManagerBean.write(DBMonElasticsearchManagerBean.java:186) at sun.reflect.GeneratedMethodAccessor1530.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1052) at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1124) at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:5388) at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:619) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800) at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571) at com.singularity.ee.controller.beans.EntityManagerInterceptor.manageEntityManagerLifecycle(EntityManagerInterceptor.java:50) at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800) at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571) at com.singularity.ee.controller.beans.ExceptionHandlingInterceptor.handleExceptions(ExceptionHandlingInterceptor.java:60) at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800) at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doAround(SystemInterceptorProxy.java:162) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:144) at sun.reflect.GeneratedMethodAccessor88.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800) at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:370) at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:5360) at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:5348) at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:214) at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:89) at com.sun.proxy.$Proxy456.write(Unknown Source) at com.singularity.ee.controller.servlet.handlers.dbmon.WriteQueryStatRequestHandler._handleRequest(WriteQueryStatRequestHandler.java:43) at com.singularity.ee.controller.servlet.handlers.a.handleRequest(a.java:411) at com.singularity.ee.controller.servlet.ControllerAgentServlet.doPost(ControllerAgentServlet.java:114) at javax.servlet.http.HttpServlet.service(HttpServlet.java:688) at javax.servlet.http.HttpServlet.service(HttpServlet.java:770) at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1550) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161) at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231) at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:317) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195) at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:860) at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:757) at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1056) at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:229) at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137) at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104) at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90) at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79) at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54) at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59) at com.sun.grizzly.ContextTask.run(ContextTask.java:121) at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:554) at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:533) at java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9080 [localhost/127.0.0.1] failed: Connection refused at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:318) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) at com.appdynamics.analytics.shared.rest.client.utils.GenericHttpRequestBuilder.getResponse(GenericHttpRequestBuilder.java:200) ... 80 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.net.SocksSocketImpl.connect(Unknown Source) at java.net.Socket.connect(Unknown Source) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72) at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:123) ... 90 more ] at com.singularity.ee.controller.beans.ExceptionHandlingInterceptor.handleExceptions(ExceptionHandlingInterceptor.java:101) at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800) at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doAround(SystemInterceptorProxy.java:162) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:144) at sun.reflect.GeneratedMethodAccessor88.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800) at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:370) at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:5360) at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:5348) at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:214) ... 31 more |#]
Do you have an idea about this problem?
Regards,
Thomas
Hi Thomas,
It looks one of the appserver agent is trying to send data to Analytics events processor and by default the events processor that comes with Controller installation does not start automatically and we need to start manually using below command:
shell> cd <Controller_install_dir>/bin
shell> controller.bat/sh start-events-service
The errors you see in controller should not affect any controller performance apart from failing to process transaction/log analytics data sent by agent, Hope that information helps.
Regards,
Arun
I do not want to use Analytics.
How can I find out which agents tries to send data to Analytics?
I have more that 100 agents and without any informations it is hard to find to correct agent.
Regards,
Thomas
Hi Thomas,
We might need to check step wise , can you check which applications when selected from app list in screen , Analytics -> Configuration at URI controller/#/location=ANALYTICS_HOME&timeRange=last_15_minutes.BEFORE_NOW.-1.-1.15 are having Analytics checkbox "enable analytics data collection for <app name>" is selected that will tell which application for which analytics enabled and in further screens in same section you should see which BTs from which tiers have this feature enabled
If no application in analytics config screen are enabled for anaytics for any app, it means the error is raised by controller internal agent or the error could be ignored given that controller tries to connect to events service by default expecting to be up.
Hope that information helps.
Regards,
Arun
Hi Arun,
no application is enabled for analytics!
But is look that each BT (names are obfuscated) reports analytic data:
Could that be a reason for the behaviour of my controller?Should I remove all BTs?
Regards,
Thomas
Hi Thomas,
Yes we request you to try the changes and if error still raises start-events-service using controller.sh/bat file and see how it goes, Keep us posted how it goes.
Regards,
Arun