Splunk AppDynamics
×

Join the Conversation

Without signing in, you're just watching from the sidelines. Sign in or Register to connect, share, and be part of the Splunk Community.
Ask a Question

Help on instrumenting AppDynamics Java Agent into Elasticsearch

Marcos_R
New Member
‎04-26-2023 07:42 AM

Hi everyone!

Currently we are trying to instrument the Java agent of AppDynamics in a Elasticsearch running on Kubernetes.

We had a few access denied errors when the Appdynamics agent tried to monitor Elasticsearch, but we resolved most with the following policy:

grant codeBase "file:/opt/appdynamics/-" {
permission java.security.AllPermission;
permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
permission java.util.PropertyPermission "*", "read,write";
permission java.lang.RuntimePermission "*";
permission java.lang.management.ManagementPermission "monitor";
permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
};
grant {
permission "java.security.SecurityPermission" "*";
permission "java.lang.RuntimePermission" "*";
permission java.io.FilePermission "<<ALL FILES>>","read,write,delete";
permission java.net.SocketPermission "*","accept,connect,resolve,listen";
permission java.util.PropertyPermission "*", "read,write";
permission "java.lang.management.ManagementPermission" "monitor";
permission "java.lang.reflect.ReflectPermission" "*";
permission "javax.management.MBeanServerPermission" "*";
permission "javax.management.MBeanPermission" "*","*";
permission "javax.management.MBeanTrustPermission" "*";
permission java.net.NetPermission "*";
};


However, at times we have the following access denied error that we are unable to resolve:

access: access denied ("java.lang.RuntimePermission" "getClassLoader")
java.lang.Exception: Stack trace
at java.base/java.lang.Thread.dumpStack(Thread.java:1379)
at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:462)
at java.base/java.security.AccessController.checkPermission(AccessController.java:1036)
at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
at java.base/java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:2058)
at java.base/java.lang.Class.getClassLoader(Class.java:836)
at com.appdynamics.appagent/com.singularity.ee.agent.appagent.services.bciengine.transformation.AnonymousClassDefTransformer.classDefTrap(AnonymousClassDefTransformer.java:61)
at com.singularity.ee.agent.appagent.entrypoint.bciengine.AnonymousClassDefTransformerBoot.classDefTrap(AnonymousClassDefTransformerBoot.java:31)
at java.base/jdk.internal.misc.Unsafe.defineAnonymousClass(Unsafe.java:1225)
at java.base/java.lang.invoke.InnerClassLambdaMetafactory.spinInnerClass(InnerClassLambdaMetafactory.java:321)
at java.base/java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(InnerClassLambdaMetafactory.java:189)
at java.base/java.lang.invoke.LambdaMetafactory.metafactory(LambdaMetafactory.java:329)
at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:127)
at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:307)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:259)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:249)
at org.elasticsearch.painless.ScriptClassInfo.<init>(ScriptClassInfo.java:75)
at org.elasticsearch.painless.Compiler.compile(Compiler.java:210)
at org.elasticsearch.painless.PainlessScriptEngine$5.run(PainlessScriptEngine.java:420)
at org.elasticsearch.painless.PainlessScriptEngine$5.run(PainlessScriptEngine.java:416)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:391)
at org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:416)
at org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:167)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:363)
at org.elasticsearch.ingest.common.ScriptProcessor$Factory.create(ScriptProcessor.java:148)
at org.elasticsearch.ingest.common.ScriptProcessor$Factory.create(ScriptProcessor.java:90)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:402)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:372)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:316)
at org.elasticsearch.ingest.Pipeline.create(Pipeline.java:73)
at org.elasticsearch.ingest.IngestService.innerUpdatePipelines(IngestService.java:515)
at org.elasticsearch.ingest.IngestService.applyClusterState(IngestService.java:259)
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:484)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:481)
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:468)
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:419)
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:163)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:830)
access: access allowed ("java.security.SecurityPermission" "getPolicy")
access: domain that failed ProtectionDomain null
null
<no principals>
java.security.Permissions@5da5ecc6 (
)


access: access denied ("java.lang.RuntimePermission" "getClassLoader")
java.lang.Exception: Stack trace
at java.base/java.lang.Thread.dumpStack(Thread.java:1379)
at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:462)
at java.base/java.security.AccessController.checkPermission(AccessController.java:1036)
at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
at java.base/java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:2058)
at java.base/java.lang.Class.getClassLoader(Class.java:836)
at com.appdynamics.appagent/com.singularity.ee.agent.appagent.services.bciengine.transformation.AnonymousClassDefTransformer.classDefTrap(AnonymousClassDefTransformer.java:61)
at com.singularity.ee.agent.appagent.entrypoint.bciengine.AnonymousClassDefTransformerBoot.classDefTrap(AnonymousClassDefTransformerBoot.java:31)
at java.base/jdk.internal.misc.Unsafe.defineAnonymousClass(Unsafe.java:1225)
at java.base/java.lang.invoke.InnerClassLambdaMetafactory.spinInnerClass(InnerClassLambdaMetafactory.java:321)
at java.base/java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(InnerClassLambdaMetafactory.java:189)
at java.base/java.lang.invoke.LambdaMetafactory.metafactory(LambdaMetafactory.java:329)
at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:127)
at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:307)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:259)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:249)
at org.elasticsearch.painless.ScriptClassInfo.<init>(ScriptClassInfo.java:86)
at org.elasticsearch.painless.Compiler.compile(Compiler.java:210)
at org.elasticsearch.painless.PainlessScriptEngine$5.run(PainlessScriptEngine.java:420)
at org.elasticsearch.painless.PainlessScriptEngine$5.run(PainlessScriptEngine.java:416)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:391)
at org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:416)
at org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:167)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:363)
at org.elasticsearch.ingest.common.ScriptProcessor$Factory.create(ScriptProcessor.java:148)
at org.elasticsearch.ingest.common.ScriptProcessor$Factory.create(ScriptProcessor.java:90)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:402)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:372)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:316)
at org.elasticsearch.ingest.Pipeline.create(Pipeline.java:73)
at org.elasticsearch.ingest.IngestService.innerUpdatePipelines(IngestService.java:515)
at org.elasticsearch.ingest.IngestService.applyClusterState(IngestService.java:259)
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:484)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:481)
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:468)
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:419)
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:163)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:830)
access: access allowed ("java.security.SecurityPermission" "getPolicy")
access: domain that failed ProtectionDomain null
null
<no principals>
java.security.Permissions@5da5ecc6 (
)


access: access denied ("java.lang.RuntimePermission" "getClassLoader")
java.lang.Exception: Stack trace
at java.base/java.lang.Thread.dumpStack(Thread.java:1379)
at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:462)
at java.base/java.security.AccessController.checkPermission(AccessController.java:1036)
at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
at java.base/java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:2058)
at java.base/java.lang.Class.getClassLoader(Class.java:836)
at com.appdynamics.appagent/com.singularity.ee.agent.appagent.services.bciengine.transformation.AnonymousClassDefTransformer.classDefTrap(AnonymousClassDefTransformer.java:61)
at com.singularity.ee.agent.appagent.entrypoint.bciengine.AnonymousClassDefTransformerBoot.classDefTrap(AnonymousClassDefTransformerBoot.java:31)
at java.base/jdk.internal.misc.Unsafe.defineAnonymousClass(Unsafe.java:1225)
at java.base/java.lang.invoke.InnerClassLambdaMetafactory.spinInnerClass(InnerClassLambdaMetafactory.java:321)
at java.base/java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(InnerClassLambdaMetafactory.java:189)
at java.base/java.lang.invoke.LambdaMetafactory.metafactory(LambdaMetafactory.java:329)
at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:127)
at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:307)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:259)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:249)
at org.elasticsearch.painless.ScriptClassInfo.methodArgument(ScriptClassInfo.java:180)
at org.elasticsearch.painless.ScriptClassInfo.<init>(ScriptClassInfo.java:99)
at org.elasticsearch.painless.Compiler.compile(Compiler.java:210)
at org.elasticsearch.painless.PainlessScriptEngine$5.run(PainlessScriptEngine.java:420)
at org.elasticsearch.painless.PainlessScriptEngine$5.run(PainlessScriptEngine.java:416)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:391)
at org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:416)
at org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:167)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:363)
at org.elasticsearch.ingest.common.ScriptProcessor$Factory.create(ScriptProcessor.java:148)
at org.elasticsearch.ingest.common.ScriptProcessor$Factory.create(ScriptProcessor.java:90)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:402)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:372)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:316)
at org.elasticsearch.ingest.Pipeline.create(Pipeline.java:73)
at org.elasticsearch.ingest.IngestService.innerUpdatePipelines(IngestService.java:515)
at org.elasticsearch.ingest.IngestService.applyClusterState(IngestService.java:259)
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:484)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:481)
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:468)
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:419)
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:163)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:830)
access: access allowed ("java.security.SecurityPermission" "getPolicy")
access: domain that failed ProtectionDomain null
null
<no principals>
java.security.Permissions@5da5ecc6 (
)

When we access the AppDynamics dashboard, we see that Elasticsearch appears online, but the only metrics captured are CPU and memory usage.

Has anyone experienced this problem or instrumented AppDynamics another way, or can you help solve and try to understand this access denied error?

PS:
- The x-pack-security is currently enabled;
- The AppDynamics Java agent is stored in a volume attached for each Elasticsearch node with read and write access;
- We tried to give access to all this access denied error;
- The java policy we created were applied successfully;
- There is no AppDynamics logs in it's workspace about this access denied error;

Labels (1)
Labels
0 Karma
Reply

Sunil_Agarwal
Communicator
‎04-27-2023 08:53 PM

Hi @Marcos.R ,

You can add below snippets to the policy , save the file and restart the ElasticSearch JVM.

permission java.lang.RuntimePermission "getClassLoader";



Reply

Marcos_R
New Member
‎05-02-2023 01:49 AM

Hi @Sunil.Agarwal,

Thank you for your answer!

I've already tried to add this permission, but the error remains.

0 Karma
Reply
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

SOC4Kafka - New Kafka Connector Powered by OpenTelemetry

The new SOC4Kafka connector, built on OpenTelemetry, enables the collection of Kafka messages and forwards ...

Event Series: Level up your SOC: Advancing with Splunk Enterprise Security

AI has fundamentally raised the stakes for security operations, and this three-part series is your guide to ...

Announcing Modern Navigation: A New Era of Splunk User Experience

We are excited to introduce the Modern Navigation feature in the Splunk Platform, available to both cloud and ...