The Java agent runs inside the application server JVM, so there is inevitably some overhead. The AppDynamics Dev team has spent years minimizing this overhead and allowing customers to dynamically adjust the balance between collecting enough information to diagnose issues vs. collecting too much information and adding excessive overhead. For example, you mentioned the call graph providing method-level performance metrics across many major packages. The agent does this without instrumenting every method. In fact, typcially fewer than 1% of all application methods are instrumented.
You're also correct that collecting diagnostic snapshots is the highest overhead activity. This is why, by default, diagnostic snapshots are collected only for Error, Slow, Very Slow, and Stalled requests, plus a sample of one snapshot per Business Transaction every 10 minutes.
This means that the vast majority of transaction executions are monitored, but no diagnostic snapshot is collected.
Yes you can tune the number of snapshots collected by configuring Thresholds: https://docs.appdynamics.com/display/PRO43/Transaction+Thresholds
It's common for performance teams to configure the Thresholds to minimize overhead in Production environment, while enabling more frequent snapshot collection in Pre-Prod environments.
