Splunk AppDynamics
×

Are you a member of the Splunk Community?

Sign in or Register with your Splunk account to get your questions answered, access valuable resources and connect with experts!
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Ask a Question

AppAgent interrupted in reporting

CommunityUser
Splunk Employee
Splunk Employee
‎10-18-2018 02:52 AM

Hello,

we using since 1 month the controller in version 4.5.0.25224 and the app agent in version 4.5.2.23891 with oracle java 1.8.0 update 141. MachineAgent is version 4.5.1.1385.

We can observe that about 10 hours after application start we receive some stacktraces and the reporting is interrupted. In this time the controller counts less calls than it is in real. After restarting the application all is fine.

[https-jsse-nio-8543-exec-7] 18 Oct 2018 11:28:17,440  WARN AgentErrorProcessor - Agent error occurred, [name,transformId]=[com.singularity.tm.ServletInterceptor - java.util.ConcurrentModificationException,367]
[https-jsse-nio-8543-exec-7] 18 Oct 2018 11:28:17,440  WARN AgentErrorProcessor - 4 instance(s) remaining before error log is silenced
[https-jsse-nio-8543-exec-7] 18 Oct 2018 11:28:17,440  WARN AgentErrorProcessor - 499 instance(s) remaining before instrumentation point is targeted for neutralization
[https-jsse-nio-8543-exec-7] 18 Oct 2018 11:28:17,440 ERROR ServletInterceptor - Error in TEP : onMethodBeginTracked for : MethodExecutionEnvironment{
 invokedObject='{ Class='org.springframework.web.servlet.DispatcherServlet', Hash code=1274906611 }',
 className='org.springframework.web.servlet.FrameworkServlet',
 methodName='doPost',
 paramValues=[ { Class='org.apache.catalina.connector.RequestFacade', Hash code=1829916242 },
{ Class='javax.servlet.http.HttpServletResponseWrapper', Hash code=678606228 } ]}, transactionContext local copy: null, transactionContext fetched from btContext unknown
java.util.ConcurrentModificationException
	at java.util.ArrayList.sort(ArrayList.java:1456) ~[?:1.8.0_141]
	at java.util.Collections.sort(Collections.java:175) ~[?:1.8.0_141]
	at com.singularity.ee.util.collections.f.a(f.java:175) ~[appagent-boot.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.services.transactionmonitor.common.cf.a(cf.java:93) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.services.transactionmonitor.common.ad.a(ad.java:424) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.services.transactionmonitor.common.ad.a(ad.java:308) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.services.transactionmonitor.common.ad.a(ad.java:252) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.services.transactionmonitor.http.servlet.k.a(k.java:68) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.services.transactionmonitor.common.i.a(i.java:1142) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.services.transactionmonitor.common.i.b(i.java:267) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.services.transactionmonitor.common.i.a(i.java:163) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.services.transactionmonitor.http.servlet.a.a(a.java:129) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.services.bciengine.a.onMethodBegin(a.java:52) ~[appagent-boot.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.kernel.bootimpl.FastMethodInterceptorDelegatorImpl.safeOnMethodBeginNoReentrantCheck(FastMethodInterceptorDelegatorImpl.java:370) ~[appagent-boot.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.kernel.bootimpl.FastMethodInterceptorDelegatorImpl.safeOnMethodBegin(FastMethodInterceptorDelegatorImpl.java:295) ~[appagent-boot.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at com.singularity.ee.agent.appagent.entrypoint.bciengine.FastMethodInterceptorDelegatorBoot.safeOnMethodBegin(FastMethodInterceptorDelegatorBoot.java:50) ~[?:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java) ~[?:?]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:648) ~[servlet-api-2.5-6.1.3.jar:6.1.3]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) ~[?:?]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) ~[servlet-api-2.5-6.1.3.jar:6.1.3]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
	at de.codecentric.accessshop.interceptor.SlotResponseHeaderFilter.doFilterInternal(SlotResponseHeaderFilter.java:31) ~[?:?]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
	at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105) ~[?:?]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
	at de.codecentric.accessshop.interceptor.LoggingServletFilter.doFilter(LoggingServletFilter.java:97) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[?:?]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
	at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89) ~[?:?]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) ~[?:?]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) ~[?:?]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
	at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:107) ~[?:?]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) ~[?:?]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108) ~[?:?]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) ~[?:?]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) ~[?:?]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) ~[?:?]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) ~[?:?]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349) ~[?:?]
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:784) ~[?:?]
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) ~[?:?]
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802) ~[?:?]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1410) ~[?:?]
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java) [?:1.8.0_141]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[?:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
[AD Thread Pool-Global189] 18 Oct 2018 11:46:21,715  INFO ErrorProcessor - Sending ADDs to register [ApplicationDiagnosticData{key='org.springframework.web.client.HttpClientErrorException:-1798508104', name=org.springframework.web.client.HttpClientErrorException:, diagnosticType=STACK_TRACE, configEntities=[Type:ERROR, id:103710], summary='org.springframework.web.client.HttpClientErrorException:'}]
[AD Thread Pool-Global189] 18 Oct 2018 11:46:21,724  WARN ErrorProcessor - ADD org.springframework.web.client.HttpClientErrorException:-1798508104 has been flagged for blacklisting. Registration will not be re-attempted until agent reset.
[AD Thread Pool-Global189] 18 Oct 2018 11:46:21,724  WARN ErrorProcessor - ADD org.springframework.web.client.HttpClientErrorException: failed to register w/ the Controller
[AD Thread Pool-Global189] 18 Oct 2018 11:46:21,724  INFO ErrorProcessor - Stack Trace Objects not registered due to controller limit: [org.springframework.web.client.HttpClientErrorException:-1798508104]

Do you have any idea how we can fix that?

with kind regards,

Raphael

Labels (1)
Labels
0 Karma
Reply

CommunityUser
Splunk Employee
Splunk Employee
‎10-19-2018 11:26 AM

Hi @Anonymous

Thank you for reporting the issue on the AppDynamics Community.

This is a known issue and our engineering team is working to fix this.

0 Karma
Reply

CommunityUser
Splunk Employee
Splunk Employee
‎10-21-2018 02:16 PM

Hi @Anonymous,

is there a workaround?

Maybe just downgrade the AppAgent to a 4.4 version?

with kind regards,

Raphael

0 Karma
Reply
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Take the Survey

Can’t make it to .conf25? Join us online!

Watch Global Broadcast
Get Updates on the Splunk Community!

Take Action Automatically on Splunk Alerts with Red Hat Ansible Automation Platform

 Are you ready to revolutionize your IT operations? As digital transformation accelerates, the demand for ...

Calling All Security Pros: Ready to Race Through Boston?

Hey Splunkers, .conf25 is heading to Boston and we’re kicking things off with something bold, competitive, and ...

Beyond Detection: How Splunk and Cisco Integrated Security Platforms Transform ...

Financial services organizations face an impossible equation: maintain 99.9% uptime for mission-critical ...