AppAgent interrupted in reporting

[CommunityUser]

Hello,

we using since 1 month the controller in version 4.5.0.25224 and the app agent in version 4.5.2.23891 with oracle java 1.8.0 update 141. MachineAgent is version 4.5.1.1385.

We can observe that about 10 hours after application start we receive some stacktraces and the reporting is interrupted. In this time the controller counts less calls than it is in real. After restarting the application all is fine.

[https-jsse-nio-8543-exec-7] 18 Oct 2018 11:28:17,440  WARN AgentErrorProcessor - Agent error occurred, [name,transformId]=[com.singularity.tm.ServletInterceptor - java.util.ConcurrentModificationException,367]
[https-jsse-nio-8543-exec-7] 18 Oct 2018 11:28:17,440  WARN AgentErrorProcessor - 4 instance(s) remaining before error log is silenced
[https-jsse-nio-8543-exec-7] 18 Oct 2018 11:28:17,440  WARN AgentErrorProcessor - 499 instance(s) remaining before instrumentation point is targeted for neutralization
[https-jsse-nio-8543-exec-7] 18 Oct 2018 11:28:17,440 ERROR ServletInterceptor - Error in TEP : onMethodBeginTracked for : MethodExecutionEnvironment{
 invokedObject='{ Class='org.springframework.web.servlet.DispatcherServlet', Hash code=1274906611 }',
 className='org.springframework.web.servlet.FrameworkServlet',
 methodName='doPost',
 paramValues=[ { Class='org.apache.catalina.connector.RequestFacade', Hash code=1829916242 },
{ Class='javax.servlet.http.HttpServletResponseWrapper', Hash code=678606228 } ]}, transactionContext local copy: null, transactionContext fetched from btContext unknown
java.util.ConcurrentModificationException
	at java.util.ArrayList.sort(ArrayList.java:1456) ~[?:1.8.0_141]
	at java.util.Collections.sort(Collections.java:175) ~[?:1.8.0_141]
	at com.singularity.ee.util.collections.f.a(f.java:175) ~[appagent-boot.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.services.transactionmonitor.common.cf.a(cf.java:93) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.services.transactionmonitor.common.ad.a(ad.java:424) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.services.transactionmonitor.common.ad.a(ad.java:308) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.services.transactionmonitor.common.ad.a(ad.java:252) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.services.transactionmonitor.http.servlet.k.a(k.java:68) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.services.transactionmonitor.common.i.a(i.java:1142) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.services.transactionmonitor.common.i.b(i.java:267) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.services.transactionmonitor.common.i.a(i.java:163) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.services.transactionmonitor.http.servlet.a.a(a.java:129) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.services.bciengine.a.onMethodBegin(a.java:52) ~[appagent-boot.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.kernel.bootimpl.FastMethodInterceptorDelegatorImpl.safeOnMethodBeginNoReentrantCheck(FastMethodInterceptorDelegatorImpl.java:370) ~[appagent-boot.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.kernel.bootimpl.FastMethodInterceptorDelegatorImpl.safeOnMethodBegin(FastMethodInterceptorDelegatorImpl.java:295) ~[appagent-boot.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.entrypoint.bciengine.FastMethodInterceptorDelegatorBoot.safeOnMethodBegin(FastMethodInterceptorDelegatorBoot.java:50) ~[?:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java) ~[?:?]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:648) ~[servlet-api-2.5-6.1.3.jar:6.1.3]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) ~[?:?]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) ~[servlet-api-2.5-6.1.3.jar:6.1.3]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
	at de.codecentric.accessshop.interceptor.SlotResponseHeaderFilter.doFilterInternal(SlotResponseHeaderFilter.java:31) ~[?:?]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
	at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105) ~[?:?]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
	at de.codecentric.accessshop.interceptor.LoggingServletFilter.doFilter(LoggingServletFilter.java:97) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[?:?]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
	at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89) ~[?:?]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) ~[?:?]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) ~[?:?]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
	at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:107) ~[?:?]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) ~[?:?]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108) ~[?:?]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) ~[?:?]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) ~[?:?]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) ~[?:?]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) ~[?:?]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349) ~[?:?]
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:784) ~[?:?]
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) ~[?:?]
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802) ~[?:?]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1410) ~[?:?]
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java) [?:1.8.0_141]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[?:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
[AD Thread Pool-Global189] 18 Oct 2018 11:46:21,715  INFO ErrorProcessor - Sending ADDs to register [ApplicationDiagnosticData{key='org.springframework.web.client.HttpClientErrorException:-1798508104', name=org.springframework.web.client.HttpClientErrorException:, diagnosticType=STACK_TRACE, configEntities=[Type:ERROR, id:103710], summary='org.springframework.web.client.HttpClientErrorException:'}]
[AD Thread Pool-Global189] 18 Oct 2018 11:46:21,724  WARN ErrorProcessor - ADD org.springframework.web.client.HttpClientErrorException:-1798508104 has been flagged for blacklisting. Registration will not be re-attempted until agent reset.
[AD Thread Pool-Global189] 18 Oct 2018 11:46:21,724  WARN ErrorProcessor - ADD org.springframework.web.client.HttpClientErrorException: failed to register w/ the Controller
[AD Thread Pool-Global189] 18 Oct 2018 11:46:21,724  INFO ErrorProcessor - Stack Trace Objects not registered due to controller limit: [org.springframework.web.client.HttpClientErrorException:-1798508104]

Do you have any idea how we can fix that?

with kind regards,

Raphael

[CommunityUser]

Hi @Anonymous

Thank you for reporting the issue on the AppDynamics Community.

This is a known issue and our engineering team is working to fix this.

[CommunityUser]

Hi @Anonymous,

is there a workaround?

Maybe just downgrade the AppAgent to a 4.4 version?

with kind regards,

Raphael