connection time out error (splunk4jmx script)

[Conradj]

Hi,

I installed and configured the Splunk4JMX but am having issues in getting a connection to my servers via jmx.

My servers are configured by the following java_opts
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.port=7337
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=true
-Dcom.sun.management.jmxremote.password=true

jmx usernames and passwords are stored under the JRE and are valid for readonly and control access.

I can access the jmx interface locally on one of the tomcat servers using jconsole via :7337 username password

but i can't access the jmx interface of the target server from my splunk server using jconsole. the connection attempts for a few seconds and then gives up.

we have internal firewalls and the servers are in different zones. A rule to allow the splunk server to connect to the tomcat server over the jmx port has been enabled and tested ok (via telnet).

worth noting too that the splunk deployment and data channel are configured the same way with firewall rules, so it doesn't appear to a firewall issue.

Prior to the firewall rule being in place the error was:
2012-01-12 09:48:38,658 ERROR [Thread-1] host=, jmxServiceURL=, jmxport=7337, jvmDescription=, processID=0,systemErrorMessage="Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: ; nested exception is:
java.net.ConnectException: Connection timed out: connect]"

with the firewall rule now inplace the error is":
2012-01-12 12:22:09,879 ERROR [Thread-1] host=, jmxServiceURL=, jmxport=7337, jvmDescription=, processID=0,systemErrorMessage="Connection refused to host: ; nested exception is:
java.net.ConnectException: Connection timed out: connect"

Can I increase the logging to debug to see if there is any addtional useful information? How would i do this?

C.

[lalit_mohan]

Thanks for your reply...

I found alternate solution from your community for above problem..

On the Universal Forwarder side of the deployment you should have these files/folders :
SplunkUniversalForwarder/etc/apps/SPLUNK4JMX/bin/ SplunkUniversalForwarder/etc/apps/SPLUNK4JMX/default/inputs.conf SplunkUniversalForwarder/etc/apps/SPLUNK4JMX/default/app.conf SplunkUniversalForwarder/etc/apps/SPLUNK4JMX/logs SplunkUniversalForwarder/etc/apps/SPLUNK4JMX/local
Then update your JMX configuration in SplunkUniversalForwarder/etc/apps/SPLUNK4JMX/bin/config.xml
Then manually enable the poll script in SplunkUniversalForwarder/etc/apps/SPLUNK4JMX/default/inputs.con*

[lalit_mohan]

Hi Guys,

I got similar error in Splunk4JMX App
I have created two vm's on microsoft azure environment set one as splunk server and one as jmx remote server.

I am getting below error on splunk server Splunk4JMX dashboard although I have configured everything.But not able to connect to this remote jmx server.

Connection refused to host: splunkserver.cloudapp.net; nested exception is: java.net.ConnectException: Connection timed out

I dont know whether it is environment issue or not.

Please help me to find the root cause.

Thanks in advance!!!

[Damien_Dallimor]

As above , seems like it is most likely a firewall issue.

[Damien_Dallimor]

Basically , the JMX RMI connector opens two ports, a multi step handshaking process.

There is the initial connection to the RMI registry on port 7337, this is the one you specify.

And then there is another port(dynamically allocated) used by the remote rmi stub , this stub performs the remoting logic used to obtain a reference to the MBeanServer, query MBeans, execute operations etc...

For the most part you don't need to worry about this second port , it is handled under the hood.

But things can get troublesome if you are behind a firewall that blocks access to random ports !

So this is more than likely what you are encountering.

There are a few workarounds , but in lieu of opening up your firewall or customizing the Tomcat JMX Agent, maybe you could try using Splunk4JMX in a Universal Forwarder(UF) that is locally running with the Tomcat instance(ergo, no firewall ports to worry about). I know several folks using this topology.And this same UF could also be used to monitor your Tomcat log files 🙂

[Damien_Dallimor]

Its a deployment scenario I have done many times,
Using deployment server to push splunk4jmx to a UF.
Exactly the same install steps as for splunk4jmx on an indexer, just need the additional outputs.conf with the forwarding logic.
If you email me direct I will help you out if you get stuck.

Splunk4JMX does also support mx4j http connectors, so only 1 http port to worry about.
But you would also need to enable this at the tomcat end .

[Conradj]

Hi,

Thanks for the speedy reply. I had a sinking feeling it was a trying to do something with a random port.

With running the tool on a universalforwarder that install is the same as for an indexer?

My installation relies exclusively on apps deployed from a deployment server. Do you have any notes that could assist in converting such an install into a deployment server friendly version?