topic Re: Getting error in tomcat with installation of appDynamics java agent in linux in AppD Archive

Getting error in tomcat with installation of appDynamics java agent in linux

CommunityUser — Tue, 16 Dec 2014 06:50:21 GMT

I am getting following error after installing appDynamics Java agent in linux server. Please let me know if anyone has seen this error and possible fix. Thanks in advance.

java.lang.IllegalStateException: getWriter() has already been called for this response
	org.apache.catalina.connector.Response.getOutputStream(Response.java:604)
	org.apache.catalina.connector.ResponseFacade.getOutputStream(ResponseFacade.java:197)
	org.springframework.http.server.ServletServerHttpResponse.getBody(ServletServerHttpResponse.java:84)
	org.springframework.http.converter.json.MappingJackson2HttpMessageConverter.writeInternal(MappingJackson2HttpMessageConverter.java:241)
	org.springframework.http.converter.AbstractHttpMessageConverter.write(AbstractHttpMessageConverter.java:207)
	org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:148)
	org.springframework.web.servlet.mvc.method.annotation.HttpEntityMethodProcessor.handleReturnValue(HttpEntityMethodProcessor.java:125)
	org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:71)
	org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:122)
	org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:749)
	org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:689)
	org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:83)
	org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:938)
	org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870)
	org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961)
	org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:863)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
	org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:728)

Re: Getting error in tomcat with installation of appDynamics java agent in linux

Arun_Dasetty — Tue, 16 Dec 2014 07:39:51 GMT

Hi Praveen,

We see similar in other customer environment when End user Monitoring(EUEM) feature enabled for business transaction request, Can you check if you have enabled EUM by an chance and if yes then it could be the case that your framework does not allow to call getWriter() twice, If this is not the case, please send the tomcat catalina and agent logs and exported app xml file using option from UI , application dashboard -> Actions -> Export

refer : http://www.coderanch.com/t/625280/Servlets/java/java-lang-IllegalStateException-getWriter-called

and we understood that agent is configured as per docs https://docs.appdynamics.com/display/PRO40/Apache+Tomcat+Startup+Settings and agent starts fine and reporting metrics but have issue with one page request.

Let us know if that information helps.

Regards,

Arun

Re: Getting error in tomcat with installation of appDynamics java agent in linux

CommunityUser — Tue, 16 Dec 2014 08:38:54 GMT

The End User Monitoring is not enabled and the Java agent is set up exactly as given in the tomcat startup settings. Appdynamics is working and I am getting data in the controller, but the application is not working with above error. Attached the app settings xml file from the controller.

Re: Getting error in tomcat with installation of appDynamics java agent in linux

Arun_Dasetty — Tue, 16 Dec 2014 14:36:17 GMT

Hi Praveen,

Somehow we do not locate any file attached, please do send the exported app xml and also the tomcat console logs / screenshot from browser tab and agent logs from <agent_install_dir>/logs path from affected JVM for our review

Regards,

Arun

Re: Getting error in tomcat with installation of appDynamics java agent in linux

CommunityUser — Wed, 17 Dec 2014 05:19:59 GMT

Attaching the logs of Tomcat as well as logs from Appdynamics Agent folder.

Re: Getting error in tomcat with installation of appDynamics java agent in linux

CommunityUser — Wed, 17 Dec 2014 09:51:36 GMT

Don't know why attachments are not coming. I tried to attach fron the page and also from email. trying one more time from different browser.

Re: Getting error in tomcat with installation of appDynamics java agent in linux

CommunityUser — Wed, 17 Dec 2014 09:56:15 GMT

Trying with zip file instead of individual files.

Re: Getting error in tomcat with installation of appDynamics java agent in linux

Arun_Dasetty — Wed, 17 Dec 2014 14:28:26 GMT

Hi,

We see from your saas ui that not all BTs are affected as per old load and could see few other BTs with agent reported fine but looks particular requests having uri segment /rest have issues , let us know if any disconnect

And we see the following in logs:

[AD Thread Pool-Global0] 16 Dec 2014 01:14:12,954 INFO ErrorProcessor - Sending ADDs to register [ApplicationDiagnosticData{key='org.springframework.web.util.NestedServletException:java.lang.IllegalStateException:-1927350763', name=org.springframework.web.util.NestedServletException:java.lang.IllegalStateException:, diagnosticType=STACK_TRACE, configEntities=[Type:ERROR, id:1323142], summary='org.springframework.web.util.NestedServletException:java.lang.IllegalStateException:'}]
..[AD Thread Pool-Global0] 16 Dec 2014 01:14:13,024 INFO ErrorProcessor - Adding entry to errorKeyToUniqueKeyMap [1323322], StackTraceErrorKey{hashCode=-1927350763}, org.springframework.web.util.NestedServletException:java.lang.IllegalStateException:-1927350763
..[AD Thread Pool-Global0] 16 Dec 2014 01:17:46,568 WARN NewTransactionWatchman - End Transaction was not invoked, cleaning up timedout transaction with :[ requestGuid [3485a5c2-78ea-481c-a538-eeeedbe5b9f6], btID [1052758], callerChain [Component From[9631] Thread Chains [1323127], Exit Type [null] To [null]], snapshotSeqString [1], snapshotEnabled [false], skewAdjustedTimestamp [null], checkAndDisableDownStreamTxDetection [false], asyncTimestamp [null] ]

Other BTs from same node no error:

Error details:

As stated earlier can you please check with your Dev team whether somehow getWriter api is called multiple times for this particular BT requests Or whether the framework does not allow to write output stream twice for the api involved?

Above said is evident from below snippet of error stacktrace from snapshot guid link : https://adppe.saas.appdynamics.com/controller/#/location=APP_SNAPSHOT_VIEWER&timeRange=last_1_week.BEFORE_NOW.-1.-1.10080&application=2192&rsdTime=Custom_Time_Range.BETWEEN_TIMES.1418636007185.1418634207185.0&requestGUID=75052457-a73f-460d-be03-eeeb531924e7 :

from BCT.logs:

[Tue Dec 16 01:10:47 PST 2014] Matching class name org/apache/catalina/connector/Response Matching class bytecode. : Class match found.
..Matching method getWriter (()Ljava/io/PrintWriter;)
[Tue Dec 16 01:12:35 PST 2014] Matching class name org/apache/catalina/connector/ResponseFacade Matching class bytecode. : Class match found.
Matching method <init> ((Lorg/apache/catalina/connector/Response;)V)//
Matching method getWriter (()Ljava/io/PrintWriter;)

Caused by: java.lang.IllegalStateException : java.lang.IllegalStateException: getWriter() has already been called for this response
at org.apache.catalina.connector.Response.getOutputStream(Response.java:604)
at org.apache.catalina.connector.ResponseFacade.getOutputStream(ResponseFacade.java:197)

ref links for more details on error: http://stackoverflow.com/questions/11305563/cause-of-servlets-response-already-committed

http://stackoverflow.com/questions/15392423/java-lang-illegalstateexception-on-response-sendredirectlocation

Let us know if that information helps.

Regards,

Arun

Re: Getting error in tomcat with installation of appDynamics java agent in linux

CommunityUser — Wed, 17 Dec 2014 14:39:11 GMT

The rest API is working without any errors when appDynamics is disabled. the error is coming only when appDynamics agent is active. So the app team is saying appD is causing the issue and asking me to disable it. This is an important application and lot of other applications use this API. So we cann't have it with errors for extended amount of time.

Re: Getting error in tomcat with installation of appDynamics java agent in linux

Arun_Dasetty — Thu, 18 Dec 2014 15:10:04 GMT

Hi,

We understood your concerns, We see similar case in other customer environment, Can you see how it goes with node property on affect node from controller UI then restart server/jvm and see how it goes:

Property name: eum-disable-filter-injection
data type: boolean
set to true and save changes.

Regards,

Arun

Re: Getting error in tomcat with installation of appDynamics java agent in linux

CommunityUser — Fri, 19 Dec 2014 10:54:59 GMT

We can't change anything on these systems up to next week as there are some tests going on these environment. I will let you know once these are completed and I get a chance to check these settings.

Re: Getting error in tomcat with installation of appDynamics java agent in linux

CommunityUser — Mon, 05 Jan 2015 07:16:51 GMT

I am not able to find the property eum-disable-filter-injection. Can you please let me know the steps how and where to set this property.

Re: Getting error in tomcat with installation of appDynamics java agent in linux

Arun_Dasetty — Mon, 05 Jan 2015 07:31:36 GMT

Hi,

Refer doc for steps to add new property from controller UI: https://docs.appdynamics.com/display/PRO39/App+Agent+Node+Properties#AppAgentNodeProperties-register-node-propertyToregisteranodeproperty , see how it goes after saving node property changes and agent jvm restart.

Regards,

Arun

Re: Getting error in tomcat with installation of appDynamics java agent in linux

CommunityUser — Mon, 05 Jan 2015 10:35:01 GMT

I added eum-disable-filter-injection property to the node. Still we are getting same error for that API call.

Re: Getting error in tomcat with installation of appDynamics java agent in linux

Arun_Dasetty — Tue, 06 Jan 2015 12:36:44 GMT

Hi,

Can you please add the following entry under <bci-processing-excludes> section in file at path /opt/appdynamics/appagent/conf/app-agent-config.xml
and see how it goes with new load post jvm restart and issue exists please send the latest archive of conf and logs folder under /opt/appdynamics/appagent/ directory

<custom-exclude filter-type="EQUALS" filter-value="org.apache.catalina.connector.ResponseFacade"/>

Suggested above changes as we see getWriter() is listed twice in your BCT.logs from above additional which is not exists in our local BCT.logs

Regards,

Arun

Re: Getting error in tomcat with installation of appDynamics java agent in linux

CommunityUser — Wed, 07 Jan 2015 06:51:18 GMT

This setting is working. I will enable this on more servers and test for some time.

Thanks

Praveen

Re: Getting error in tomcat with installation of appDynamics java agent in linux

Arun_Dasetty — Wed, 07 Jan 2015 07:06:23 GMT

Hi Praveen,

Glad to hear that provided suggestions did the trick, keep us posted how it goes on other nodes.

Regards,

Arun