JVM deadlock after splunk-library-javalogging upgrade from 1.6.1 to 1.11.1

[Dipabhat]

Hi Team,

Our application is having a jvm deadlock and stopping indefinetly after splunk-library-javalogging upgrade from 1.6.1 to 1.11.1.
Below is the logback, slf4j versions used.

compile 'ch.qos.logback:logback-classic:1.2.3'
compile 'ch.qos.logback:logback-core:1.2.3'
compile 'org.slf4j:slf4j-api:1.7.36'
compile 'net.logstash.logback:logstash-logback-encoder:6.6'

compile "uk.org.lidalia:sysout-over-slf4j:1.0.2"

And below is error deadlock thread stack

default task-340	Name[default task-340]Thread ID[13667] Deadlocked on Lock[ch.qos.logback.core.spi.LogbackLock@31309b21] held by thread [splunk-tcp-appender] Thread ID[209] Thread stack [ ch.qos.logback.core.BasicStatusManager.fireStatusAddEvent(BasicStatusManager.java:85) ch.qos.logback.core.BasicStatusManager.add(BasicStatusManager.java:59) ch.qos.logback.core.spi.ContextAwareBase.addStatus(ContextAwareBase.java:80) ch.qos.logback.core.spi.ContextAwareBase.addInfo(ContextAwareBase.java:85) com.splunk.logging.TcpAppender.append(TcpAppender.java:291) com.splunk.logging.TcpAppender.append(TcpAppender.java:40) ch.qos.logback.core.AppenderBase.doAppend(AppenderBase.java:82) ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51) ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270) ch.qos.logback.classic.Logger.callAppenders(Logger.java:257) ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421) ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383) ch.qos.logback.classic.Logger.info(Logger.java:579) uk.org.lidalia.sysoutslf4j.context.LogLevel$3.log(LogLevel.java:62) uk.org.lidalia.sysoutslf4j.context.LoggerAppenderImpl.log(LoggerAppenderImpl.java:81) uk.org.lidalia.sysoutslf4j.context.LoggerAppenderImpl.logOrPrint(LoggerAppenderImpl.java:71) uk.org.lidalia.sysoutslf4j.context.LoggerAppenderImpl.appendAndLog(LoggerAppenderImpl.java:58) uk.org.lidalia.sysoutslf4j.system.SLF4JPrintStreamDelegate.appendAndLog(SLF4JPrintStreamDelegate.java:76) uk.org.lidalia.sysoutslf4j.system.SLF4JPrintStreamDelegate.delegatePrintln(SLF4JPrintStreamDelegate.java:56) uk.org.lidalia.sysoutslf4j.system.SLF4JPrintStreamImpl.println(SLF4JPrintStreamImpl.java:111) com.lehman.elmo.admin.mvc.controller.ELMRestController.getFolder(ELMRestController.java:120) sun.reflect.GeneratedMethodAccessor116.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067) org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) javax.servlet.http.HttpServlet.service(HttpServlet.java:503) org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) javax.servlet.http.HttpServlet.service(HttpServlet.java:590) io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) com.lehman.admin.servlet.EtgAdminServletFilter.doFilter(EtgAdminServletFilter.java:141) io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67) io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67) io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:94) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67) io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) com.lehman.elmo.admin.SimpleCORSFilter.doFilter(SimpleCORSFilter.java:22) io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67) io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) com.lehman.elmo.admin.ui.filter.ElmoRootFilter.doFilter(ElmoRootFilter.java:94) io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67) io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) io.undertow.servlet.handlers.Filte
default task-355	Name[default task-355]Thread ID[14086] Deadlocked on Lock[com.splunk.logging.TcpAppender@7a29da17] held by thread [default task-340] Thread ID[13667] Thread stack [ ch.qos.logback.core.AppenderBase.doAppend(AppenderBase.java:63) ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51) ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270) ch.qos.logback.classic.Logger.callAppenders(Logger.java:257) ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421) ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383) ch.qos.logback.classic.Logger.log(Logger.java:765) org.apache.logging.slf4j.SLF4JLogger.logMessage(SLF4JLogger.java:234) org.apache.logging.log4j.spi.AbstractLogger.log(AbstractLogger.java:2117) org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2205) org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2159) org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2142) org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2017) org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1983) org.apache.logging.log4j.spi.AbstractLogger.info(AbstractLogger.java:1320) com.lehman.elmo.admin.ui.filter.ElmoRootFilter.doFilter(ElmoRootFilter.java:90) io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67) io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117) io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) io.undertow.server.handlers.DisableCacheHandler.handleRequest(DisableCacheHandler.java:33) io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53) io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:60) io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68) io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52) io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:64) io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59) io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:275) io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:79) io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:134) io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:131) io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.jav
splunk-tcp-appender	Name[splunk-tcp-appender]Thread ID[209] Deadlocked on Lock[uk.org.lidalia.sysoutslf4j.system.SLF4JPrintStreamImpl@5a3259b] held by thread [default task-340] Thread ID[13667] Thread stack [ uk.org.lidalia.sysoutslf4j.system.SLF4JPrintStreamImpl.print(SLF4JPrintStreamImpl.java:246) ch.qos.logback.core.status.OnPrintStreamStatusListenerBase.print(OnPrintStreamStatusListenerBase.java:52) ch.qos.logback.core.status.OnPrintStreamStatusListenerBase.addStatusEvent(OnPrintStreamStatusListenerBase.java:58) ch.qos.logback.core.BasicStatusManager.fireStatusAddEvent(BasicStatusManager.java:87) ch.qos.logback.core.BasicStatusManager.add(BasicStatusManager.java:59) ch.qos.logback.core.spi.ContextAwareBase.addStatus(ContextAwareBase.java:80) ch.qos.logback.core.spi.ContextAwareBase.addInfo(ContextAwareBase.java:85) com.splunk.logging.TcpAppender.run(TcpAppender.java:130) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) ]