If you are trying to instrument Payara with AppD, you need to first disable the built in monitoring in Payara. You can do this easily through the Payara admin panel. Just uncheck the "monitoring" check box. Then you can follow the instructions in our docs for instrumenting Glassfish.
Enjoy.
Thanks, @Allan.Schiebold for the tip.