SSL/TLS with requireClientCert in web.conf fails

[kaurinko]

Hi!

I have worked for a while to make Splunk use TLS and PKI as much as possible. At present the system contains of version 8.0.1 components only. I have managed to get Splunk Indexer to require client certificate from the UFs, and it seems to work. The Splunk serverCert is in a file containing the certificate, private key, issuer certificate and the root-CA certificate (the issuer of the issuer certificate). The trusted root-CA -certificates are in a separate text file. For this connection requiring requireClientCert = true works fine.

With the web-UI things are not going as elegantly. My web.conf looks like this:

[settings]
enableSplunkWebSSL = 1
privKeyPath = etc/auth/splunkweb/splunk.pki.key
serverCert = etc/auth/splunkweb/splunk.pki.txt
requireClientCert = false
sslVersions = tls1.2
loginBackgroundImageOption = none
login_content = This is a <b>Test installation</b>.

With the above, things are just fine. Changing requireClientCert to true breaks everything and the webGUI is not started. The splunkd.log gets populated with lines like:

03-09-2020 12:45:04.223 +0200 ERROR X509Verify - X509 certificate (CN=Root CA,O=X,C=Y) failed validation; error=19, reason="self signed certificate in certificate chain"

I get exactly the same error message, if I connect using openSSL to the indexer port, where UFs connect:

openssl s_client -connect splunk:9998 -state -prexit
*
Certificate chain
0 s:/C=Y/O=X/CN=Test Splunk Indexer
  i:/C=Y/O=X/CN=TestCA-1
1 s:/C=Y/O=X/CN=TestCA-1
  i:/C=Y/O=X/CN=Root CA
2 s:/C=Y/O=X/CN=Root CA
  i:/C=Y/O=X/CN=Root CA
*
SSL-Session:
   Protocol  : TLSv1.2
*
   Verify return code: 19 (self signed certificate in certificate chain)

My point here is that the connection between UF and indexer still works fine. The question is: Is requireClientCertificate simply not supported in the web-GUI, or is there something in the documentation I have not understood correctly?

If it is possible to require a certificate from the client (i.e. a web browser), is there a way to define the trusted CA-certificates and should any intermediate CA certifictes be included as well?

Another thing I have been wondering about is certificate validation and CRLs. Is there a way to make Splunk actually validate the certificates it is presented?

Best regards,

Petri

[harsmarvania57 · Accepted solution]

Hi,

Here few things that differ from UF -> Indexer TLS connectivity and User -> Splunk Web TLS connectivity.

When you configure serverCert in web.conf, it does not require private key in certification chain.

serverCert = <path>
* Full path to the Privacy Enhanced Mail (PEM) format Splunk web server certificate file.
* The file may also contain root and intermediate certificates, if required.
  They should be listed sequentially in the order:
    [ Server SSL certificate ]
    [ One or more intermediate certificates, if required ]
    [ Root certificate, if required ]
* See also 'enableSplunkWebSSL' and 'privKeyPath'.
* Default: $SPLUNK_HOME/etc/auth/splunkweb/cert.pem

Second if your private key is protected with password then you need to configure sslPassword in web.conf OR remove password from private key and configure it, in that case you do not need to configure sslPassword in web.conf

sslPassword = <password>
* Password that protects the private key specified by 'privKeyPath'.
* If encrypted private key is used, do not enable client-authentication
  on splunkd server. In [sslConfig] stanza of server.conf,
  'requireClientCert' must be 'false'.
* Optional.
* Default: The unencrypted private key.

And when you configure requireClientCert=true in web.conf you need to configure sslRootCAPath = <PATH> in server.conf

web.conf

requireClientCert = <boolean>
* Requires that any HTTPS client that connects to the Splunk Web HTTPS
  server has a certificate that was signed by the CA cert installed
  on this server.
* If "true", a client can connect ONLY if a certificate created by our
  certificate authority was used on that client.
* If "true", it is mandatory to configure splunkd with same root CA in server.conf.
  This is needed for internal communication between splunkd and splunkweb.
* Default: false

[kaurinko]

Hi,

That nailed it, thanks! I had to add the Root-CA certificate of the splunkweb-cert to the file pointed at by sslRootCAPath and obviously all the other trusted Root-CAs.

The documentation for this part is confusing. With certificates the keyword I am looking for is trust. The documentation is not clear about how to define the trusted CAs (trust anchors) for different purposes. Actually the wording lets one believe that there can only be one trusted CA, which would be very strange. Now that I know how things work, I can understand it, but it should have been the other way around.

Do you know anything about doing proper certificate validation with CRLs or an ocsp in Splunk?

/Petri

[harsmarvania57]

You can concatenate multiple root CA in single Root CA file and configure that root CA file in server.conf . For CRL don't you think that is browser's responsibility to check whether certificate is revoked or not ?

[kaurinko]

I think it is the responsibility of any relying party to proper validation. That would include checking that the certificate is valid, signed by the issuer, not revoked and the issuer certificate is also valid. That last step adds recursion to the process, but most chains are not that long.

I would say that the browser should be interested in validating the server certificate, but if a client certificate is requested and presented, it would be the server's duty to validate it.

[harsmarvania57]

As far as I am aware CRL signing perfomed by browser, splunk do not support oscp as of now (I can't see any setting to enable this) and for client side verification you can verify client certificate using CN

web.conf

sslCommonNameToCheck = <commonName1>, <commonName2>, ...
* Checks the common name of the client's certificate against this list of names.
* 'requireClientCert' must be set to "true" for this setting to work.
* Optional. 
* Default: empty string (No common name checking).

[kaurinko]

I guess you got mixed somewhere writing your comment, or I can't follow the logic "CRL signing by browser" doesn't make any sense as CRLs are signed by CAs. Anyhow, my guess is Splunk isn't interested in CRLs as I haven't found anything about them in the documentation. The other thing I can't find mentioned is validity period checking, so it looks like the certificates are treated as never expiring certificates.

Despite all that, I still think setting up certificates for the connections is worth the trouble. And maybe some day there will be better validation.

[harsmarvania57]

Yes it got mixed up for CRL, I mean to say CRL verification. For any features which are not available now but you think that those will good one then you can submit your feature request on https://ideas.splunk.com

[kaurinko]

I'll think about that. Thanks for the conversation.

[harsmarvania57]

Welcome 🙂