Help on instrumenting AppDynamics Java Agent into Elasticsearch

[Marcos_R]

Hi everyone!

Currently we are trying to instrument the Java agent of AppDynamics in a Elasticsearch running on Kubernetes.

We had a few access denied errors when the Appdynamics agent tried to monitor Elasticsearch, but we resolved most with the following policy:

grant codeBase "file:/opt/appdynamics/-" {
permission java.security.AllPermission;
permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
permission java.util.PropertyPermission "*", "read,write";
permission java.lang.RuntimePermission "*";
permission java.lang.management.ManagementPermission "monitor";
permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
};
grant {
permission "java.security.SecurityPermission" "*";
permission "java.lang.RuntimePermission" "*";
permission java.io.FilePermission "<<ALL FILES>>","read,write,delete";
permission java.net.SocketPermission "*","accept,connect,resolve,listen";
permission java.util.PropertyPermission "*", "read,write";
permission "java.lang.management.ManagementPermission" "monitor";
permission "java.lang.reflect.ReflectPermission" "*";
permission "javax.management.MBeanServerPermission" "*";
permission "javax.management.MBeanPermission" "*","*";
permission "javax.management.MBeanTrustPermission" "*";
permission java.net.NetPermission "*";
};

However, at times we have the following access denied error that we are unable to resolve:

access: access denied ("java.lang.RuntimePermission" "getClassLoader")
java.lang.Exception: Stack trace
at java.base/java.lang.Thread.dumpStack(Thread.java:1379)
at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:462)
at java.base/java.security.AccessController.checkPermission(AccessController.java:1036)
at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
at java.base/java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:2058)
at java.base/java.lang.Class.getClassLoader(Class.java:836)
at com.appdynamics.appagent/com.singularity.ee.agent.appagent.services.bciengine.transformation.AnonymousClassDefTransformer.classDefTrap(AnonymousClassDefTransformer.java:61)
at com.singularity.ee.agent.appagent.entrypoint.bciengine.AnonymousClassDefTransformerBoot.classDefTrap(AnonymousClassDefTransformerBoot.java:31)
at java.base/jdk.internal.misc.Unsafe.defineAnonymousClass(Unsafe.java:1225)
at java.base/java.lang.invoke.InnerClassLambdaMetafactory.spinInnerClass(InnerClassLambdaMetafactory.java:321)
at java.base/java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(InnerClassLambdaMetafactory.java:189)
at java.base/java.lang.invoke.LambdaMetafactory.metafactory(LambdaMetafactory.java:329)
at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:127)
at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:307)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:259)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:249)
at org.elasticsearch.painless.ScriptClassInfo.<init>(ScriptClassInfo.java:75)
at org.elasticsearch.painless.Compiler.compile(Compiler.java:210)
at org.elasticsearch.painless.PainlessScriptEngine$5.run(PainlessScriptEngine.java:420)
at org.elasticsearch.painless.PainlessScriptEngine$5.run(PainlessScriptEngine.java:416)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:391)
at org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:416)
at org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:167)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:363)
at org.elasticsearch.ingest.common.ScriptProcessor$Factory.create(ScriptProcessor.java:148)
at org.elasticsearch.ingest.common.ScriptProcessor$Factory.create(ScriptProcessor.java:90)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:402)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:372)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:316)
at org.elasticsearch.ingest.Pipeline.create(Pipeline.java:73)
at org.elasticsearch.ingest.IngestService.innerUpdatePipelines(IngestService.java:515)
at org.elasticsearch.ingest.IngestService.applyClusterState(IngestService.java:259)
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:484)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:481)
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:468)
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:419)
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:163)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:830)
access: access allowed ("java.security.SecurityPermission" "getPolicy")
access: domain that failed ProtectionDomain null
null
<no principals>
java.security.Permissions@5da5ecc6 (
)


access: access denied ("java.lang.RuntimePermission" "getClassLoader")
java.lang.Exception: Stack trace
at java.base/java.lang.Thread.dumpStack(Thread.java:1379)
at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:462)
at java.base/java.security.AccessController.checkPermission(AccessController.java:1036)
at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
at java.base/java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:2058)
at java.base/java.lang.Class.getClassLoader(Class.java:836)
at com.appdynamics.appagent/com.singularity.ee.agent.appagent.services.bciengine.transformation.AnonymousClassDefTransformer.classDefTrap(AnonymousClassDefTransformer.java:61)
at com.singularity.ee.agent.appagent.entrypoint.bciengine.AnonymousClassDefTransformerBoot.classDefTrap(AnonymousClassDefTransformerBoot.java:31)
at java.base/jdk.internal.misc.Unsafe.defineAnonymousClass(Unsafe.java:1225)
at java.base/java.lang.invoke.InnerClassLambdaMetafactory.spinInnerClass(InnerClassLambdaMetafactory.java:321)
at java.base/java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(InnerClassLambdaMetafactory.java:189)
at java.base/java.lang.invoke.LambdaMetafactory.metafactory(LambdaMetafactory.java:329)
at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:127)
at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:307)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:259)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:249)
at org.elasticsearch.painless.ScriptClassInfo.<init>(ScriptClassInfo.java:86)
at org.elasticsearch.painless.Compiler.compile(Compiler.java:210)
at org.elasticsearch.painless.PainlessScriptEngine$5.run(PainlessScriptEngine.java:420)
at org.elasticsearch.painless.PainlessScriptEngine$5.run(PainlessScriptEngine.java:416)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:391)
at org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:416)
at org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:167)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:363)
at org.elasticsearch.ingest.common.ScriptProcessor$Factory.create(ScriptProcessor.java:148)
at org.elasticsearch.ingest.common.ScriptProcessor$Factory.create(ScriptProcessor.java:90)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:402)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:372)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:316)
at org.elasticsearch.ingest.Pipeline.create(Pipeline.java:73)
at org.elasticsearch.ingest.IngestService.innerUpdatePipelines(IngestService.java:515)
at org.elasticsearch.ingest.IngestService.applyClusterState(IngestService.java:259)
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:484)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:481)
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:468)
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:419)
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:163)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:830)
access: access allowed ("java.security.SecurityPermission" "getPolicy")
access: domain that failed ProtectionDomain null
null
<no principals>
java.security.Permissions@5da5ecc6 (
)


access: access denied ("java.lang.RuntimePermission" "getClassLoader")
java.lang.Exception: Stack trace
at java.base/java.lang.Thread.dumpStack(Thread.java:1379)
at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:462)
at java.base/java.security.AccessController.checkPermission(AccessController.java:1036)
at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
at java.base/java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:2058)
at java.base/java.lang.Class.getClassLoader(Class.java:836)
at com.appdynamics.appagent/com.singularity.ee.agent.appagent.services.bciengine.transformation.AnonymousClassDefTransformer.classDefTrap(AnonymousClassDefTransformer.java:61)
at com.singularity.ee.agent.appagent.entrypoint.bciengine.AnonymousClassDefTransformerBoot.classDefTrap(AnonymousClassDefTransformerBoot.java:31)
at java.base/jdk.internal.misc.Unsafe.defineAnonymousClass(Unsafe.java:1225)
at java.base/java.lang.invoke.InnerClassLambdaMetafactory.spinInnerClass(InnerClassLambdaMetafactory.java:321)
at java.base/java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(InnerClassLambdaMetafactory.java:189)
at java.base/java.lang.invoke.LambdaMetafactory.metafactory(LambdaMetafactory.java:329)
at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:127)
at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:307)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:259)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:249)
at org.elasticsearch.painless.ScriptClassInfo.methodArgument(ScriptClassInfo.java:180)
at org.elasticsearch.painless.ScriptClassInfo.<init>(ScriptClassInfo.java:99)
at org.elasticsearch.painless.Compiler.compile(Compiler.java:210)
at org.elasticsearch.painless.PainlessScriptEngine$5.run(PainlessScriptEngine.java:420)
at org.elasticsearch.painless.PainlessScriptEngine$5.run(PainlessScriptEngine.java:416)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:391)
at org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:416)
at org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:167)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:363)
at org.elasticsearch.ingest.common.ScriptProcessor$Factory.create(ScriptProcessor.java:148)
at org.elasticsearch.ingest.common.ScriptProcessor$Factory.create(ScriptProcessor.java:90)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:402)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:372)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:316)
at org.elasticsearch.ingest.Pipeline.create(Pipeline.java:73)
at org.elasticsearch.ingest.IngestService.innerUpdatePipelines(IngestService.java:515)
at org.elasticsearch.ingest.IngestService.applyClusterState(IngestService.java:259)
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:484)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:481)
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:468)
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:419)
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:163)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:830)
access: access allowed ("java.security.SecurityPermission" "getPolicy")
access: domain that failed ProtectionDomain null
null
<no principals>
java.security.Permissions@5da5ecc6 (
)

When we access the AppDynamics dashboard, we see that Elasticsearch appears online, but the only metrics captured are CPU and memory usage.

Has anyone experienced this problem or instrumented AppDynamics another way, or can you help solve and try to understand this access denied error?

PS:
- The x-pack-security is currently enabled;
- The AppDynamics Java agent is stored in a volume attached for each Elasticsearch node with read and write access;
- We tried to give access to all this access denied error;
- The java policy we created were applied successfully;
- There is no AppDynamics logs in it's workspace about this access denied error;

[Sunil_Agarwal]

Hi @Marcos.R ,

You can add below snippets to the policy , save the file and restart the ElasticSearch JVM.

permission java.lang.RuntimePermission "getClassLoader"; 

[Marcos_R]

Hi @Sunil.Agarwal,

Thank you for your answer!

I've already tried to add this permission, but the error remains.