since I enabled the setting "requireClientCert = true" on our server.conf files the App "SplunkTAjmx" just stopped working. I pasted the error messages at the end.
Once the setting is returned to "false" the app starts working again.
We are using selfsigned Certificates on our Splunk to Splunk communications, apart from this App, all other connections are working perfectly with requireClientCert = true .
I even tried generating the file mx4j.ks. No success. 😞
I seems the App internal connection to splunk are being blocked, but I can't find a way to provide it with out certificates.
Any recommendation? Is it a bug?
We are running on SLES 11, Splunk 6.2.2 build 255606. Splunk Add-on for Java Management Extensions 3.0.0 (sandbox version is 3.0.1) and Oracle Java 1.8.
Thanks in advance,
on splunkd.log 06-26-2015 15:09:37.491 +0200 WARN HttpListener - Socket error from 127.0.0.1 while idling: error:140890C7:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:peer did not return a certificate on jmx.log 2015-06-26 14:26:09,630 - com.splunk.modinput.ModularInput -0 [main] ERROR - Error executing modular input : Received fatal alert: handshake_failure : java.lang.RuntimeException: Received fatal alert: handshake_failure
Java 8 means TLS is required, SSL won't work. I'd also upgrade to the latest Splunkd.
That is not the issue here. A simple test with Java 1.8 and openssl s_server using the same certificates from my Splunk system returns successfull connections. This error can be reproduced by not sending a client certificate.
Btw. an updated version of splunk (sandbox running 6.2.3) returns the same problem.
allow me to clarify -- this is the tested and supported connectivity matrix: http://docs.splunk.com/Documentation/AddOns/latest/JMX/Hardwareandsoftwarerequirements#Prerequisites
If you're trying to go outside of that, we don't think that it will work, but will happily accept being wrong if it comes with a support ticket and enhancement request, preferably with an example of how it was made to work 🙂
We did not try to go outside of that. A Support ticket has just been opened (Case Nr. 251396).
If you wish to see it, just edit the [sslConfig] stanza of your server.conf as follows (alter paths and filenames as necessary) (either on a splunk server of a universal forwarder)
Once its done, restart splunk and check your splunkd.log and jmx.log files.
[sslConfig] allowSslCompression = false allowSslRenegotiation = false caCertFile = <self_signed_root_ca> caPath = <caPath> cipherSuite = TLSv1+HIGH:!SSLv2:!RC2:!RC4:!DES:!3DES:!MD5:!MD2:!EXP:!MEDIUM:!LOW:!PSK:!DSS:!aNULL:!eNULL:!SRP:!aECDH:!aECDSA@STRENGTH ecdhCurveName = prime256v1 requireClientCert = true sslKeysfile = <sslKeysfile> sslKeysfilePassword = <sslKeysfilePassword> sslVersions = tls1.2
A simple test with Java 1.8 and openssl s_server using the same certificates from my Splunk system returns successfull connections.
Just for clarification; this means you have your certs imported in the Java keystore that is used by the app as well, correct?
No for the following causes:
1- because there is no mention of it on the Installation steps ( http://docs.splunk.com/Documentation/AddOns/latest/JMX/Installationsteps)
2- The only point of conflict is the variable in question: requireClientCert = true (if it is set to off, the app starts working).
For me it is pretty clear that some improvement is due on this App.
Any solution or workaround for this problem without setting requireClientCert = false
here the official answer I got from Splunk support:
"Unfortunately the feedback from Dev is that JMX App does not support requireClientCert=true in server.conf.
They are planning to add the fix the one of the next releases of this App, so I would like to know if using requireClientCert=false it is a possibility based on your requirements"
So we have to work with requireClientCert=false for the time being.