Monitoring of Java Virtual Machines with JMX: Why am I receiving "Connection refused to host" error?

[ShaneNewman]

I am running into an issue when trying to connect to Alfresco/Solr using the Monitoring of Java Virtual Machines with JMX app and the Splunk app for Alfresco. I believe I have the settings correct (checked it a few times and got some other eyes on it). I am getting these error messages, which aren't very helpful.

10-12-2016 22:08:39.576 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py" host=myhost, jmxServiceURL=, jmxport=50500, jvmDescription=alfresco, processID=0,stanza=jmx://Alfresco,systemErrorMessage="Connection refused to host: x.x.x.x; nested exception is:

10-12-2016 22:08:39.576 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py"    java.net.ConnectException: Connection refused"

I have verified that I am able to telnet to the host from the Splunk Heavy Forwarder (where this is setup) to the alfresco server on port 50500 without issue. Username and Password verified as well.

This is a snippet from the alfresco-config.xml that might be useful with sensitive info removed:

<jmxserver host="myhost" jvmDescription="alfresco" jmxport="50500" jmxuser="controlRole" jmxpass="mypass" lookupPath="/alfresco/jmxrmi">

Telnet to confirm no network issues:

[splunk@splunk-util02 jmxconfig]$ telnet myhost 50500
Trying x.x.x.x...
Connected to myhost.

Any assistance trying to get this figured out would be greatly appreciated!

[Damien_Dallimor]

java.net.ConnectException: Connection refused

Firewall perhaps. Telnet is a pointless test because RMI works differently ie: other ports get dynamically opened after the initial handshake.

Can you connect using JConsole from the same machine where you are running the JMX App ?

[ShaneNewman]

This is RHEL 7, no UI. Are there commands that I can run from CLI to interact with JConsole? I was told by the Alfresco team that the only port that is listening for JMX connections is 50500. Is this not the case? Should I have additional ports opened up on the JMX host?

From JMX Host:
tcp 0 0 *:50500 * : * LISTEN

[Damien_Dallimor]

Other ports are unknown.Hence I wrote "dynamically".
Try JConsole from some other machine where there is a GUI.

Furthermore , just from googling and reading docs (I don't know alfresco) , http://docs.alfresco.com/5.0/tasks/jmx-access.html

In particular point 4. It looks like a custom JMX URL is needed ,which you can declare in the jmxServiceURL field

service:jmx:rmi:///jndi/rmi://<hostname>:50500/alfresco/jmxrmi

[ShaneNewman]

Also, firewall/iptables are disabled on this host (Splunk host).

[ShaneNewman]

Perhaps important to note, I have also tried to use the IP instead of the FQDN which has the same results.