I am not able to Start the controller..Please help!!!

[CommunityUser]

Hi,

I am trying to start my controller but its not coming up and below is the exception which i am encountering from server.log and bottom is the output from database.log.I am afraid some might have mistakenly deleted some files.Please advise/help ASAP. I am using 3.8.x.x version.

Server.log--

#|2014-10-06T15:35:23.859-0400|SEVERE|glassfish3.1.2|javax.enterprise.system.tools.deployment.org.glassfish.deployment.common|_ThreadID=1;_ThreadName=Thread-6;|Exception while invoking class org.glassfish.ejb.startup.EjbDeployer load method
java.lang.RuntimeException: EJB Container initialization error
 at org.glassfish.ejb.startup.EjbApplication.loadContainers(EjbApplication.java:242)
 at org.glassfish.ejb.startup.EjbDeployer.load(EjbDeployer.java:305)
 at org.glassfish.ejb.startup.EjbDeployer.load(EjbDeployer.java:108)
 at org.glassfish.internal.data.ModuleInfo.load(ModuleInfo.java:186)
 at org.glassfish.internal.data.ApplicationInfo.load(ApplicationInfo.java:264)
 at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:460)
 at com.sun.enterprise.v3.server.ApplicationLoaderService.processApplication(ApplicationLoaderService.java:375)
 at com.sun.enterprise.v3.server.ApplicationLoaderService.postConstruct(ApplicationLoaderService.java:219)
 at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:131)
 at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:91)
 at com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:82)
 at com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:67)
 at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:139)
 at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:78)
 at com.sun.enterprise.v3.server.AppServerStartup.run(AppServerStartup.java:253)
 at com.sun.enterprise.v3.server.AppServerStartup.doStart(AppServerStartup.java:145)
 at com.sun.enterprise.v3.server.AppServerStartup.start(AppServerStartup.java:136)
 at com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.start(GlassFishImpl.java:79)
 at com.sun.enterprise.glassfish.bootstrap.GlassFishDecorator.start(GlassFishDecorator.java:63)
 at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishImpl.start(OSGiGlassFishImpl.java:69)
 at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:117)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:97)
 at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55)
Caused by: java.lang.Exception
 at com.sun.enterprise.connectors.inbound.ConnectorMessageBeanClient.setup(ConnectorMessageBeanClient.java:233)
 at com.sun.ejb.containers.MessageBeanContainer.<init>(MessageBeanContainer.java:205)
 at com.sun.ejb.containers.ContainerFactoryImpl.createContainer(ContainerFactoryImpl.java:121)
 at org.glassfish.ejb.startup.EjbApplication.loadContainers(EjbApplication.java:230)
 ... 26 more
Caused by: java.lang.NullPointerException
 at com.sun.messaging.jms.ra.DirectConnectionFactory._createConnectionId(DirectConnectionFactory.java:424)
 at com.sun.messaging.jms.ra.DirectConnectionFactory._createConnection(DirectConnectionFactory.java:522)
 at com.sun.messaging.jms.ra.EndpointConsumer.createDirectMessageConsumer(EndpointConsumer.java:752)
 at com.sun.messaging.jms.ra.EndpointConsumer._init(EndpointConsumer.java:306)
 at com.sun.messaging.jms.ra.EndpointConsumer.<init>(EndpointConsumer.java:185)
 at com.sun.messaging.jms.ra.ResourceAdapter.endpointActivation(ResourceAdapter.java:478)
 at com.sun.enterprise.connectors.inbound.ConnectorMessageBeanClient.setup(ConnectorMessageBeanClient.java:225)
 ... 29 more

Database.log==>#|2014-10-06T15:35:23.859-0400|SEVERE|glassfish3.1.2|javax.enterprise.system.tools.deployment.org.glassfish.deployment.common|_ThreadID=1;_ThreadName=Thread-6;|Exception while invoking class org.glassfish.ejb.startup.EjbDeployer load method
java.lang.RuntimeException: EJB Container initialization error
 at org.glassfish.ejb.startup.EjbApplication.loadContainers(EjbApplication.java:242)
 at org.glassfish.ejb.startup.EjbDeployer.load(EjbDeployer.java:305)
 at org.glassfish.ejb.startup.EjbDeployer.load(EjbDeployer.java:108)
 at org.glassfish.internal.data.ModuleInfo.load(ModuleInfo.java:186)
 at org.glassfish.internal.data.ApplicationInfo.load(ApplicationInfo.java:264)
 at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:460)
 at com.sun.enterprise.v3.server.ApplicationLoaderService.processApplication(ApplicationLoaderService.java:375)
 at com.sun.enterprise.v3.server.ApplicationLoaderService.postConstruct(ApplicationLoaderService.java:219)
 at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:131)
 at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:91)
 at com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:82)
 at com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:67)
 at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:139)
 at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:78)
 at com.sun.enterprise.v3.server.AppServerStartup.run(AppServerStartup.java:253)
 at com.sun.enterprise.v3.server.AppServerStartup.doStart(AppServerStartup.java:145)
 at com.sun.enterprise.v3.server.AppServerStartup.start(AppServerStartup.java:136)
 at com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.start(GlassFishImpl.java:79)
 at com.sun.enterprise.glassfish.bootstrap.GlassFishDecorator.start(GlassFishDecorator.java:63)
 at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishImpl.start(OSGiGlassFishImpl.java:69)
 at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:117)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:97)
 at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55)
Caused by: java.lang.Exception
 at com.sun.enterprise.connectors.inbound.ConnectorMessageBeanClient.setup(ConnectorMessageBeanClient.java:233)
 at com.sun.ejb.containers.MessageBeanContainer.<init>(MessageBeanContainer.java:205)
 at com.sun.ejb.containers.ContainerFactoryImpl.createContainer(ContainerFactoryImpl.java:121)
 at org.glassfish.ejb.startup.EjbApplication.loadContainers(EjbApplication.java:230)
 ... 26 more
Caused by: java.lang.NullPointerException
 at com.sun.messaging.jms.ra.DirectConnectionFactory._createConnectionId(DirectConnectionFactory.java:424)
 at com.sun.messaging.jms.ra.DirectConnectionFactory._createConnection(DirectConnectionFactory.java:522)
 at com.sun.messaging.jms.ra.EndpointConsumer.createDirectMessageConsumer(EndpointConsumer.java:752)
 at com.sun.messaging.jms.ra.EndpointConsumer._init(EndpointConsumer.java:306)
 at com.sun.messaging.jms.ra.EndpointConsumer.<init>(EndpointConsumer.java:185)
 at com.sun.messaging.jms.ra.ResourceAdapter.endpointActivation(ResourceAdapter.java:478)
 at com.sun.enterprise.connectors.inbound.ConnectorMessageBeanClient.setup(ConnectorMessageBeanClient.java:225)
 ... 29 more

[Arun_Dasetty]

Hi,

Can you attach latest server.log, startAs.log and database.log to debug further?

While you attach the latest couple of server.log and database.log under <Controller_install_dir>/logs path (seems database.log content pasted is from server.log itself in initial post)  Can you see if the following information helps:

- check if there is another imq message process running on port 7676 and having lock on files under path <controller_dir>/appserver/glassfish/domains/domain1/imq/instances/imqbroker/lock

- kill all jave process and delete any existing locked files and do a stepwise restart of controler db mysql and glassfish process and see if that helps

Regards,

Arun

[CommunityUser]

Hi Arun,

PFA the log files and the startAS.log only had one line "Attempting to start domain1.... Please look at the server log for more details....."

Which are attached now. Also I tried killing the java process and tried to restart but it didn't workout.

Also do you think some of the files may be missing or got corrupted under "\AppDynamics\Controller\db\data\controller" which is causing this issue??

[Arun_Dasetty]

Hi,

Somehow we could not locate any of the logs in attachments, please attach the logs requested to assist you faster.

Regarding your other query, We would need to check database.log to see if any corrupted db table files are causing any issue here.

We see earlier similar issue and we suspect the imq message broker file lock could be creating issue, but we need to recent couple of server.log and other logs to confirm the same.

Can you check the output of the following command and also see the imq file if exists at path stated earlier:

shell> ps -ef | grep java

Make sure no java process running and try to re-run controller startup commands again.

Regards,

Arun

[CommunityUser]

PFA the log files and also i did searched for java processes which are running and and tried to start the controller but in vain.

[Arun_Dasetty]

Hi,

We see controller is not able to start as part of disk space check and which is evident from below set of logs:

================================

[#|2014-10-06T14:04:15.015-0400|SEVERE|glassfish3.1.2|com.appdynamics.METRICS.WRITE|_ThreadID=129;_ThreadName=Thread-6;|Controller shutting down due to very low disk space!  Space left is 1018MB|#]

..

javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.3.2.v20111125-r10461): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Error in allocating a connection. Cause: Exception while destroying physical connection
Error Code: 0

================================

We also lot many table related errors in database.log and this could be due to low space and also we see mysql connection failure 

Can you peform following steps and see how it goes:

To disable automatic shutdown for a running Controller
1. Shut down the Controller. See Start or Stop the Controller
2. Open the <Controller-Installation-Directory>/appserver/glassfish/domains/domain1/config/domain.xml file for edit.
3. Add the following:
<jvm-options>-Dignore.disk.space=true</jvm-options>
4. Save the file.
5. Restart the Controller.

However we recommend to make sure your controller meets system requirements as per docs: http://docs.appdynamics.com/display/PRO14S/Controller+System+Requirements , as we see running low on disk space will result in corruption of controller db tables.

Additonally you can re-visit the controller settings that could help to purge data by reducing the retention settings as stated in doc: https://docs.appdynamics.com/display/PRO14S/Database+Size+and+Data+Retention

Let us knwo if that information helps.

Regards,

Arun

[CommunityUser]

Hi Arun,

Thanks for your reply but the disk space on controller is not low as i can see the disk have 8.4gb available on it and i am aware of the steps you mentioned in previous email and i am sure that is not the case for this issue as controller is not coming up so how can it shutdown automatically.

Also, how do we verify that all the files under this location are ok and nothing is missing?

==> :AppDynamics\Controller\db\data\controller 

Thank you.

[Arun_Dasetty]

Hi,

We understood your concern we see that there are very high chances that low space (though you said more space exists logs says otherwise somehow) will not let controller UI acessible itself as we see errors in allocation to database controlelr mysql.

Regarding your other query , we can issue the following command to check table status:

cmd> cd D:\AppDynamics\Controller\bin

cmd> controller.bat login-db

mysql> show table status \G;

Or more specific tables

mysql> show table status like '%metric%' \G;

Hope that information helps.

Regards,

Arun