Controller takes ages to start on cloud

[CommunityUser]

Hi, 

I have installed appdynamics on cloud. But it takes almost 30 minutes, just for the controller to start. My CPU Configuration is at its peak though. 28 GB ram, 8 core processor, 500 gb harddisk. 

Please advise.

[Arun_Dasetty]

Hi ,

Can you please provide the profile installed and number of agent nodes trying to report controller and also check the profile installed is meeting system reuqirements as per doc once: https://docs.appdynamics.com/display/PRO39/Controller+System+Requirements#ControllerSystemRequiremen...

If meeting system requirements send the domain.xml and server.log, database.log for review once including startAs.log

Regards,

Arun

[CommunityUser]

Hi, I am running a demo license controller. 1 machine node. 2 database node.

[Arun_Dasetty]

Hi,

Can you provide the output of the following:

- install.log from F:\AppDynamics\Controller path

- screenshot from UI Setup -> AppDynamics Agents screen

- Login controller internal app using URI http://JSSIndiaDB:8090/controller?enableAccounts=true (system/root/changeme) and provide screenshot from node dashboard -> Memory screen

- Also provide the RAM and CPU cores and disk space details of the controller instance.

We requested above details as we do not see any relevant errors in logs provided apart from startAs.log showing delay in startup

Regards,

Arun

[CommunityUser]

Processor -

ARK | Intel® Xeon® Processor E5-2660

Ram- 28 GB

Disk Space : 500 GB

[CommunityUser]

And I am also seeing error in server logs which I sent you earlier.

[Arun_Dasetty]

Hi,

The h/w config looks fine and from logs we see controller started in 10 minutes as per logs between restarts. Can you point us the error you are referring in server.log ?

While you provide above logs, Can you try following changes as we see h/w required for demo profile looks fine:

-  Stop the controller appserver, we assume that system is not overloaded on machine controller is installed,
Check the http-listener is enabled fine in :F:/AppDynamics/Controller/appserver/glassfish/domains/domain/config/domain.xml file and increase the heap size -Xms from 1024 to 2048 and Xmx to 4098m and see how it goes with restart of process?

-Xms1024m
-Xmx1024m

Regards,

Arun

[CommunityUser]

See the above zipped file named as New Cmpressed. 

[#|2014-12-04T08:40:43.882+0000|SEVERE|glassfish3.1.2|com.sun.jersey.spi.container.ContainerResponse|_ThreadID=45;_ThreadName=Thread-6;|The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
java.lang.NullPointerException
at com.singularity.ee.controller.ui.services.ErrorsUiService.getErrorSummary(ErrorsUiService.java)
at sun.reflect.GeneratedMethodAccessor927.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1550)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:343)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:217)
at com.singularity.ee.controller.servlet.CacheControlFilter.doFilter(CacheControlFilter.java)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:217)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:279)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:317)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:860)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:757)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1056)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:229)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:121)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:554)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:533)
at java.lang.Thread.run(Unknown Source)
|#]

It takes 10 minutes to start in my local. And 30 minutes to start on Cloud. And sorry I cannot stop the appServer. It is a big risk, which I think might not be the ideal solution.

[Arun_Dasetty]

Hi ,

I could confirm that error should not be an issue as i could see similar error in my local controller running on windows on 3.9.x version:

Nov 20, 2014 10:50:57 AM com.sun.enterprise.admin.launcher.GFLauncherLogger info
-Dappdynamics.controller.appserver=D:\AppDynamics\Controller395ga\appserver\glassfish
[#|2014-11-20T17:15:20.942+0530|INFO|glassfish3.1.2|com.singularity.ee.controller.ui.services.FeatureUiService|_ThreadID=57;_ThreadName=Thread-6;|Analytics enabled for account [customer1]? false|#]

[#|2014-11-20T17:15:26.407+0530|SEVERE|glassfish3.1.2|com.sun.jersey.spi.container.ContainerResponse|_ThreadID=60;_ThreadName=Thread-6;|The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
java.lang.NullPointerException
at com.singularity.ee.controller.ui.services.ErrorsUiService.getErrorSummary(ErrorsUiService.java)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

Can you please confirm the memory usage trend in local instance and cloud instance from contrller internal agent, we see earlier you have provided output not from controller internal applciation, we request you to logout controller UI and loging using /controller?enableAccoounts=true URI (system/root/changeme) as login details , where changeme is root password by default and capture memory usage for contorller on cloud

and also provide output of the following command on controller running on cloud:

shell> free -m (or command /screenshot that gives similar memory usage output)

Can you also list the company name here, we would like to check in portal whether you have permissions to raise ticket at help@appdynamics.com so that we can involve our controller experts team on this.

Regards,

Arun

[CommunityUser]

My company name is Caterpillar. And the memory usage box is empty, i have attached the file.

[Arun_Dasetty]

Hi ,

We request you to raise help ticket help.appdynamics.com Or send email to help@appdynamics.com so that we can check the behavior in web call, we understood that you could not restart controller appserver can you be more specific the reason the reason that could not restart, we thought to check the behavior with controller restart in call to observe the behavior in live also be sure that restarting controller glassfish will not affect the server with agent configured that is sending performance metrics apart from lossing performance monitoring for period controller is restarted.

Regards,

Arun

[CommunityUser]

I know that part, restarting agent will not affect my settings, I have done this thousand times. But I cannot restart now because , we are having demos round the clock, anytime. If I restart the controller and the demo comes in , i will be screwed. 

[Arun_Dasetty]

Hi ,

We understood your concerns, We request you to raise help ticket to see if other team members have any thoughts here that can be done without restart , atleast i believe that to check the behavior or changes we would need to restart which is not possible as per your demo commitments , i would suggest you to raise ticket once you are done with demo commitments and you have slot to restart so that we could check the behavior in live and will check if the delay is due to other resources are consuming CPU cycles and memory usage is at peak during the same period where restart is in progress.

Hope that plan of action is fine with you.

Regards,

Arun

[CommunityUser]

I have rasised the ticket. If you can come up to webex I can show you live demo of what is happening. Should I send you an invite? Please share your email id.

[Arun_Dasetty]

Hi,

We have sent the invite on ticket raised, thanks for raising ticket.

Regards,

Arun