All Apps and Add-ons

Monitoring of Java Virtual Machines with JMX Issue with SSL connection

mikeli621
Engager

We have setup Monitoring of Java Virtual Machines with JMX app on our application server. We are getting following error.

2022-03-03 13:27:22 INFO Logger=ModularInput Initialising Modular Input
2022-03-03 13:27:22 INFO Logger=ModularInput Getting scheme
2022-03-03 13:27:25 INFO Logger=ModularInput Initialising Modular Input
2022-03-03 13:27:25 INFO Logger=ModularInput Running connection poller
2022-03-03 13:27:26 INFO Logger=ModularInput Running state checker
2022-03-03 13:27:26 INFO Logger=ModularInput Activation key check passed
2022-03-03 13:27:26 INFO Logger=org.exolab.castor.mapping.Mapping Loading mapping descriptors from jar:file:/splunk/splunkforwarder/etc/apps/SPLUNK4JMX/bin/lib/jmxmodinput.jar!/mapping.xml
2022-03-03 13:29:33 ERROR Logger=jmx://fakevalue host=xxx.xx.xxx.xxx, jmxServiceURL=, jmxport=6969, jvmDescription=fakevalue, processID=0,stanza=jmx://xxx,systemErrorMessage="Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: xxx.xx.xxx.xxx; nested exception is:
java.net.ConnectException: Connection timed out (Connection timed out)]"

Below is a portion of our Java setting (masked). We handle some parameters in Catalina property files.

JVM_OPTS="-Dhae.cf.srv.util.security.hmacKey=fakevalue -XX:ReservedCodeCacheSize=500m -Xmx16G -Xms16G -Xss1024K -Dorg.quartz.threadPool.threadCount=1 -Dspring.security.strategy=fakevalue -Dorg.apache.el.parser.SKIP_IDENTIFIER_CHECK=true -Duser.timezone=UTC -Dhae.cf.srv.sec.timeout=15 -Dhpf.session.expiry=30 -Dng.log.dir=xxx -Djavax.net.ssl.trustStore=xxx/cacerts.jks -Djavax.net.ssl.trustStorePassword=xxxxx -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Xloggc:

I suspect the issue is with SSL configuration, and we have same deployment to non-ssl Java environment. Things are working as expected.

How can we add SSL related configuration into config.xml?

Current connection:

<jmxserver host="xxx.xx.xxx.xxx" jvmDescription="fakevalue" jmxport="6969" jmxuser="admin" jmxpass="xxxx">

@Damien_Dallimor 

Labels (1)
0 Karma
1 Solution

PickleRick
SplunkTrust
SplunkTrust

Your suspicion is, I suspect, wrong 😄

But seriously, you get an exception "connection refused" and "connection timed-out". It most probably means that there is a problem on a lower network level -mthe TCP connection is not established before there would be any TLS negotiation.

Can't tell you what it is obviously but the typical approach to debugging such situation is verifying network connectivity by hand from your forwarder and doing a dump of network traffic and looking what's going on there on IP/TCP layers. If the connection is established properly, look into TLS negotiation.

View solution in original post

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Your suspicion is, I suspect, wrong 😄

But seriously, you get an exception "connection refused" and "connection timed-out". It most probably means that there is a problem on a lower network level -mthe TCP connection is not established before there would be any TLS negotiation.

Can't tell you what it is obviously but the typical approach to debugging such situation is verifying network connectivity by hand from your forwarder and doing a dump of network traffic and looking what's going on there on IP/TCP layers. If the connection is established properly, look into TLS negotiation.

0 Karma

mikeli621
Engager

You are 100% right.

I tried this seting on other nodes belonging to the same cluster and it's working as expected.

It definitely looking like Network related issue for that application node only. Interesting other logs are flowing just JVM are seeing timeouts.

 

 

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...