CONTROLLER Cannot Login

[Tadeu_Alves]

Hi there,

Today my Controller got stuck with high CPU usage, i restarted it and now for some weird reason no one can login i can only acess the /controller/admin.jsp

Inside the server log i got a lot of entries like this:

[#|2021-12-06T22:39:37.411-0300|INFO|glassfish 4.1|javax.enterprise.system.core.security|_ThreadID=69;_ThreadName=http-listener-1(13);_TimeMillis=1638841177411;_LevelValue=800;_MessageID=NCLS-SECURITY-05046;|Audit: Authentication refused for [singularity-agent@customer1].|#]

[#|2021-12-06T22:39:37.412-0300|INFO|glassfish 4.1|javax.enterprise.system.core.security.com.sun.enterprise.security.jmac.callback|_ThreadID=69;_ThreadName=http-listener-1(13);_TimeMillis=1638841177412;_LevelValue=800;|jmac.loginfail|#]

I even tried to create a new Account but now when i log in everything is blank.. in the controller did i break something?

Is it possible to recover what i "destroyed"

Any help is appreciated.

[Tadeu_Alves]

Guys the server is "comming back" i did nothing and all of sudden some parts of the configuration are showing, but i don't see nothing of the instrumentation that i did and when i log in i get this error: 

Error
500 Internal Server Error

Exception Id:b13eec1b-bc08-463d-b77f-19f881a04c67

Looking inside the server.log i see many SEVERE errors, all of them are:

Unable to decrypt AccountAccess Key.
com.singularity.ee.util.security.credentialstore.exception.CryptoOperationFailureException: Failed to decrypt: Input length must be multiple of 16 when decrypting with padded cipher

Here is one of the many stacks that show the same error only different threads:

com.singularity.ee.controller.entity.AccountEntity|_ThreadID=1010;_ThreadName=config-caches-refresh-54;_TimeMillis=1638886343390;_LevelValue=1000;|Unable to decrypt AccountAccess Key.
com.singularity.ee.util.security.credentialstore.exception.CryptoOperationFailureException: Failed to decrypt: Input length must be multiple of 16 when decrypting with padded cipher
at com.singularity.ee.util.security.credentialstore.KeyStoreBasedEncryptionWrapper.decryptCipherTextString(KeyStoreBasedEncryptionWrapper.java:97)
at com.singularity.ee.util.scs.SCSDecryptionUtilityManager.decryptCipherText(SCSDecryptionUtilityManager.java:86)
at com.singularity.ee.controller.entity.AccountEntity.scsOperationsDoToDTO(AccountEntity.java:716)
at com.singularity.ee.controller.entity.AccountEntity.postLoadLifeCycleEvent(AccountEntity.java:705)
at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.hibernate.ejb.event.BeanCallback.invoke(BeanCallback.java:37)
at org.hibernate.ejb.event.EntityCallbackHandler.callback(EntityCallbackHandler.java:94)
at org.hibernate.ejb.event.EntityCallbackHandler.postLoad(EntityCallbackHandler.java:87)
at org.hibernate.ejb.event.EJB3PostLoadEventListener.onPostLoad(EJB3PostLoadEventListener.java:47)
at org.hibernate.event.def.DefaultLoadEventListener.assembleCacheEntry(DefaultLoadEventListener.java:700)
at org.hibernate.event.def.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:605)
at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:459)
at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:227)
at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:147)
at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:1090)
at org.hibernate.impl.SessionImpl.immediateLoad(SessionImpl.java:1026)
at org.hibernate.proxy.AbstractLazyInitializer.initialize(AbstractLazyInitializer.java:176)
at org.hibernate.proxy.AbstractLazyInitializer.getImplementation(AbstractLazyInitializer.java:215)
at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:190)
at com.singularity.ee.controller.entity.AccountEntity_$$_jvst12b_108.isMarkedAsDeleted(AccountEntity_$$_jvst12b_108.java)
at com.appdynamics.platform.apm.service.backend.BackendConfigCacheServiceImpl.refreshCache(BackendConfigCacheServiceImpl.java:380)
at com.appdynamics.platform.apm.service.backend.BackendConfigCacheServiceImpl$$EnhancerByGuice$$2ddbec59.CGLIB$refreshCache$1(<generated>)
at com.appdynamics.platform.apm.service.backend.BackendConfigCacheServiceImpl$$EnhancerByGuice$$2ddbec59$$FastClassByGuice$$8ce8ba83.invoke(<generated>)
at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:75)
at com.appdynamics.platform.persistence.TransactionInterceptor.lambda$invoke$0(TransactionInterceptor.java:37)
at com.singularity.ee.controller.beans.model.EJBManagerBean.runWithinSupportsTransaction(EJBManagerBean.java:23)
at sun.reflect.GeneratedMethodAccessor116.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4786)
at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:824)
at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
at com.appdynamics.platform.persistence.EntityManagerInterceptor.call(EntityManagerInterceptor.java:45)
at com.appdynamics.platform.persistence.GlassfishEntityManagerInterceptor.manageEntityManagerLifecycle(GlassfishEntityManagerInterceptor.java:25)
at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:823)
at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
at com.appdynamics.controller.persistence.ControllerExceptionHandlingInterceptor.call(ControllerExceptionHandlingInterceptor.java:71)
at com.appdynamics.controller.persistence.GlassfishControllerExceptionHandlingInterceptor.handleExceptions(GlassfishControllerExceptionHandlingInterceptor.java:18)
at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:823)
at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:823)
at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:369)
at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4758)
at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4746)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88)
at com.sun.proxy.$Proxy340.runWithinSupportsTransaction(Unknown Source)
at com.appdynamics.platform.persistence.GlassfishPersistenceManager.runWithinTransactionAttributeBlock(GlassfishPersistenceManager.java:35)
at com.appdynamics.platform.persistence.TransactionInterceptor.invoke(TransactionInterceptor.java:33)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:75)
at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:54)
at com.appdynamics.platform.apm.service.backend.BackendConfigCacheServiceImpl$$EnhancerByGuice$$2ddbec59.refreshCache(<generated>)
at com.singularity.ee.controller.beans.ControllerManager.lambda$getCacheRefreshForAppTasks$8(ControllerManager.java:2657)
at com.appdynamics.platform.services.threads.ExecutorRegistry$ReportMetricsCallable.call(ExecutorRegistry.java:474)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.crypto.IllegalBlockSizeException: Input length must be multiple of 16 when decrypting with padded cipher
at com.sun.crypto.provider.CipherCore.prepareInputBuffer(CipherCore.java:1005)
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:848)
at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:446)
at javax.crypto.Cipher.doFinal(Cipher.java:2168)
at com.singularity.ee.util.security.credentialstore.impl.EncryptionSchemeLegacy.decrypt(EncryptionSchemeLegacy.java:68)
at com.singularity.ee.util.security.credentialstore.EncryptionCommon.decryptCipherTextString(EncryptionCommon.java:102)
at com.singularity.ee.util.security.credentialstore.KeyStoreBasedEncryptionWrapper.decryptCipherTextString(KeyStoreBasedEncryptionWrapper.java:94)
... 77 more

[iamryan]

Hi @Tadeu.Alves,

This seems like a critical issue, I would contact Support for more immediate help. If you do and support helps you, can you please share what you learned here?

Knowledge sharing is what drives this community forward.