Monitoring of Java Virtual Machines with JMX Issue with SSL connection

[mikeli621]

We have setup Monitoring of Java Virtual Machines with JMX app on our application server. We are getting following error.

2022-03-03 13:27:22 INFO Logger=ModularInput Initialising Modular Input
2022-03-03 13:27:22 INFO Logger=ModularInput Getting scheme
2022-03-03 13:27:25 INFO Logger=ModularInput Initialising Modular Input
2022-03-03 13:27:25 INFO Logger=ModularInput Running connection poller
2022-03-03 13:27:26 INFO Logger=ModularInput Running state checker
2022-03-03 13:27:26 INFO Logger=ModularInput Activation key check passed
2022-03-03 13:27:26 INFO Logger=org.exolab.castor.mapping.Mapping Loading mapping descriptors from jar:file:/splunk/splunkforwarder/etc/apps/SPLUNK4JMX/bin/lib/jmxmodinput.jar!/mapping.xml
2022-03-03 13:29:33 ERROR Logger=jmx://fakevalue host=xxx.xx.xxx.xxx, jmxServiceURL=, jmxport=6969, jvmDescription=fakevalue, processID=0,stanza=jmx://xxx,systemErrorMessage="Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: xxx.xx.xxx.xxx; nested exception is:
java.net.ConnectException: Connection timed out (Connection timed out)]"

Below is a portion of our Java setting (masked). We handle some parameters in Catalina property files.

JVM_OPTS="-Dhae.cf.srv.util.security.hmacKey=fakevalue -XX:ReservedCodeCacheSize=500m -Xmx16G -Xms16G -Xss1024K -Dorg.quartz.threadPool.threadCount=1 -Dspring.security.strategy=fakevalue -Dorg.apache.el.parser.SKIP_IDENTIFIER_CHECK=true -Duser.timezone=UTC -Dhae.cf.srv.sec.timeout=15 -Dhpf.session.expiry=30 -Dng.log.dir=xxx -Djavax.net.ssl.trustStore=xxx/cacerts.jks -Djavax.net.ssl.trustStorePassword=xxxxx -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Xloggc:

I suspect the issue is with SSL configuration, and we have same deployment to non-ssl Java environment. Things are working as expected.

How can we add SSL related configuration into config.xml?

Current connection:

<jmxserver host="xxx.xx.xxx.xxx" jvmDescription="fakevalue" jmxport="6969" jmxuser="admin" jmxpass="xxxx">

@Damien_Dallimor 

[PickleRick · Accepted solution]

Your suspicion is, I suspect, wrong 😄

But seriously, you get an exception "connection refused" and "connection timed-out". It most probably means that there is a problem on a lower network level -mthe TCP connection is not established before there would be any TLS negotiation.

Can't tell you what it is obviously but the typical approach to debugging such situation is verifying network connectivity by hand from your forwarder and doing a dump of network traffic and looking what's going on there on IP/TCP layers. If the connection is established properly, look into TLS negotiation.

[mikeli621]

You are 100% right.

I tried this seting on other nodes belonging to the same cluster and it's working as expected.

It definitely looking like Network related issue for that application node only. Interesting other logs are flowing just JVM are seeing timeouts.