All Apps and Add-ons

Monitoring of Java Virtual Machines with JMX Issue with SSL connection

mikeli621
Engager

We have setup Monitoring of Java Virtual Machines with JMX app on our application server. We are getting following error.

2022-03-03 13:27:22 INFO Logger=ModularInput Initialising Modular Input
2022-03-03 13:27:22 INFO Logger=ModularInput Getting scheme
2022-03-03 13:27:25 INFO Logger=ModularInput Initialising Modular Input
2022-03-03 13:27:25 INFO Logger=ModularInput Running connection poller
2022-03-03 13:27:26 INFO Logger=ModularInput Running state checker
2022-03-03 13:27:26 INFO Logger=ModularInput Activation key check passed
2022-03-03 13:27:26 INFO Logger=org.exolab.castor.mapping.Mapping Loading mapping descriptors from jar:file:/splunk/splunkforwarder/etc/apps/SPLUNK4JMX/bin/lib/jmxmodinput.jar!/mapping.xml
2022-03-03 13:29:33 ERROR Logger=jmx://fakevalue host=xxx.xx.xxx.xxx, jmxServiceURL=, jmxport=6969, jvmDescription=fakevalue, processID=0,stanza=jmx://xxx,systemErrorMessage="Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: xxx.xx.xxx.xxx; nested exception is:
java.net.ConnectException: Connection timed out (Connection timed out)]"

Below is a portion of our Java setting (masked). We handle some parameters in Catalina property files.

JVM_OPTS="-Dhae.cf.srv.util.security.hmacKey=fakevalue -XX:ReservedCodeCacheSize=500m -Xmx16G -Xms16G -Xss1024K -Dorg.quartz.threadPool.threadCount=1 -Dspring.security.strategy=fakevalue -Dorg.apache.el.parser.SKIP_IDENTIFIER_CHECK=true -Duser.timezone=UTC -Dhae.cf.srv.sec.timeout=15 -Dhpf.session.expiry=30 -Dng.log.dir=xxx -Djavax.net.ssl.trustStore=xxx/cacerts.jks -Djavax.net.ssl.trustStorePassword=xxxxx -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Xloggc:

I suspect the issue is with SSL configuration, and we have same deployment to non-ssl Java environment. Things are working as expected.

How can we add SSL related configuration into config.xml?

Current connection:

<jmxserver host="xxx.xx.xxx.xxx" jvmDescription="fakevalue" jmxport="6969" jmxuser="admin" jmxpass="xxxx">

@Damien_Dallimor 

Labels (1)
0 Karma
1 Solution

PickleRick
Ultra Champion

Your suspicion is, I suspect, wrong 😄

But seriously, you get an exception "connection refused" and "connection timed-out". It most probably means that there is a problem on a lower network level -mthe TCP connection is not established before there would be any TLS negotiation.

Can't tell you what it is obviously but the typical approach to debugging such situation is verifying network connectivity by hand from your forwarder and doing a dump of network traffic and looking what's going on there on IP/TCP layers. If the connection is established properly, look into TLS negotiation.

View solution in original post

0 Karma

PickleRick
Ultra Champion

Your suspicion is, I suspect, wrong 😄

But seriously, you get an exception "connection refused" and "connection timed-out". It most probably means that there is a problem on a lower network level -mthe TCP connection is not established before there would be any TLS negotiation.

Can't tell you what it is obviously but the typical approach to debugging such situation is verifying network connectivity by hand from your forwarder and doing a dump of network traffic and looking what's going on there on IP/TCP layers. If the connection is established properly, look into TLS negotiation.

0 Karma

mikeli621
Engager

You are 100% right.

I tried this seting on other nodes belonging to the same cluster and it's working as expected.

It definitely looking like Network related issue for that application node only. Interesting other logs are flowing just JVM are seeing timeouts.

 

 

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...