CONTROLLER Cannot Login

Tadeu_Alves — Tue, 07 Dec 2021 00:48:08 GMT

Hi there,

Today my Controller got stuck with high CPU usage, i restarted it and now for some weird reason no one can login i can only acess the /controller/admin.jsp

Inside the server log i got a lot of entries like this:

I even tried to create a new Account but now when i log in everything is blank.. in the controller did i break something?

Is it possible to recover what i "destroyed"

Any help is appreciated.

Re: CONTROLLER Cannot Login

Tadeu_Alves — Tue, 07 Dec 2021 14:17:25 GMT

Guys the server is "comming back" i did nothing and all of sudden some parts of the configuration are showing, but i don't see nothing of the instrumentation that i did and when i log in i get this error:

Error
500 Internal Server Error

Exception Id:b13eec1b-bc08-463d-b77f-19f881a04c67

Looking inside the server.log i see many SEVERE errors, all of them are:

Unable to decrypt AccountAccess Key.
com.singularity.ee.util.security.credentialstore.exception.CryptoOperationFailureException: Failed to decrypt: Input length must be multiple of 16 when decrypting with padded cipher

Here is one of the many stacks that show the same error only different threads:

com.singularity.ee.controller.entity.AccountEntity|_ThreadID=1010;_ThreadName=config-caches-refresh-54;_TimeMillis=1638886343390;_LevelValue=1000;|Unable to decrypt AccountAccess Key.
com.singularity.ee.util.security.credentialstore.exception.CryptoOperationFailureException: Failed to decrypt: Input length must be multiple of 16 when decrypting with padded cipher
at com.singularity.ee.util.security.credentialstore.KeyStoreBasedEncryptionWrapper.decryptCipherTextString(KeyStoreBasedEncryptionWrapper.java:97)
at com.singularity.ee.util.scs.SCSDecryptionUtilityManager.decryptCipherText(SCSDecryptionUtilityManager.java:86)
at com.singularity.ee.controller.entity.AccountEntity.scsOperationsDoToDTO(AccountEntity.java:716)
at com.singularity.ee.controller.entity.AccountEntity.postLoadLifeCycleEvent(AccountEntity.java:705)
at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.hibernate.ejb.event.BeanCallback.invoke(BeanCallback.java:37)
at org.hibernate.ejb.event.EntityCallbackHandler.callback(EntityCallbackHandler.java:94)
at org.hibernate.ejb.event.EntityCallbackHandler.postLoad(EntityCallbackHandler.java:87)
at org.hibernate.ejb.event.EJB3PostLoadEventListener.onPostLoad(EJB3PostLoadEventListener.java:47)
at org.hibernate.event.def.DefaultLoadEventListener.assembleCacheEntry(DefaultLoadEventListener.java:700)
at org.hibernate.event.def.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:605)
at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:459)
at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:227)
at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:147)
at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:1090)
at org.hibernate.impl.SessionImpl.immediateLoad(SessionImpl.java:1026)
at org.hibernate.proxy.AbstractLazyInitializer.initialize(AbstractLazyInitializer.java:176)
at org.hibernate.proxy.AbstractLazyInitializer.getImplementation(AbstractLazyInitializer.java:215)
at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:190)
at com.singularity.ee.controller.entity.AccountEntity_$$_jvst12b_108.isMarkedAsDeleted(AccountEntity_$$_jvst12b_108.java)
at com.appdynamics.platform.apm.service.backend.BackendConfigCacheServiceImpl.refreshCache(BackendConfigCacheServiceImpl.java:380)
at com.appdynamics.platform.apm.service.backend.BackendConfigCacheServiceImpl$$EnhancerByGuice$$2ddbec59.CGLIB$refreshCache$1(<generated>)
at com.appdynamics.platform.apm.service.backend.BackendConfigCacheServiceImpl$$EnhancerByGuice$$2ddbec59$$FastClassByGuice$$8ce8ba83.invoke(<generated>)
at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:75)
at com.appdynamics.platform.persistence.TransactionInterceptor.lambda$invoke$0(TransactionInterceptor.java:37)
at com.singularity.ee.controller.beans.model.EJBManagerBean.runWithinSupportsTransaction(EJBManagerBean.java:23)
at sun.reflect.GeneratedMethodAccessor116.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4786)
at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:824)
at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
at com.appdynamics.platform.persistence.EntityManagerInterceptor.call(EntityManagerInterceptor.java:45)
at com.appdynamics.platform.persistence.GlassfishEntityManagerInterceptor.manageEntityManagerLifecycle(GlassfishEntityManagerInterceptor.java:25)
at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:823)
at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
at com.appdynamics.controller.persistence.ControllerExceptionHandlingInterceptor.call(ControllerExceptionHandlingInterceptor.java:71)
at com.appdynamics.controller.persistence.GlassfishControllerExceptionHandlingInterceptor.handleExceptions(GlassfishControllerExceptionHandlingInterceptor.java:18)
at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:823)
at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:823)
at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:369)
at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4758)
at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4746)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88)
at com.sun.proxy.$Proxy340.runWithinSupportsTransaction(Unknown Source)
at com.appdynamics.platform.persistence.GlassfishPersistenceManager.runWithinTransactionAttributeBlock(GlassfishPersistenceManager.java:35)
at com.appdynamics.platform.persistence.TransactionInterceptor.invoke(TransactionInterceptor.java:33)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:75)
at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:54)
at com.appdynamics.platform.apm.service.backend.BackendConfigCacheServiceImpl$$EnhancerByGuice$$2ddbec59.refreshCache(<generated>)
at com.singularity.ee.controller.beans.ControllerManager.lambda$getCacheRefreshForAppTasks$8(ControllerManager.java:2657)
at com.appdynamics.platform.services.threads.ExecutorRegistry$ReportMetricsCallable.call(ExecutorRegistry.java:474)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.crypto.IllegalBlockSizeException: Input length must be multiple of 16 when decrypting with padded cipher
at com.sun.crypto.provider.CipherCore.prepareInputBuffer(CipherCore.java:1005)
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:848)
at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:446)
at javax.crypto.Cipher.doFinal(Cipher.java:2168)
at com.singularity.ee.util.security.credentialstore.impl.EncryptionSchemeLegacy.decrypt(EncryptionSchemeLegacy.java:68)
at com.singularity.ee.util.security.credentialstore.EncryptionCommon.decryptCipherTextString(EncryptionCommon.java:102)
at com.singularity.ee.util.security.credentialstore.KeyStoreBasedEncryptionWrapper.decryptCipherTextString(KeyStoreBasedEncryptionWrapper.java:94)
... 77 more

Re: CONTROLLER Cannot Login

iamryan — Tue, 07 Dec 2021 17:53:37 GMT

Hi @Tadeu.Alves,

This seems like a critical issue, I would contact Support for more immediate help. If you do and support helps you, can you please share what you learned here?

Knowledge sharing is what drives this community forward.

topic CONTROLLER Cannot Login in Splunk AppDynamics

CONTROLLER Cannot Login

Re: CONTROLLER Cannot Login

Re: CONTROLLER Cannot Login