After upgrading to 6.4, why are our signed certs no longer accepted in server.conf?

[andrewcg]

After upgrading the to 6.4, Splunk web would no longer start:

Starting splunk server daemon (splunkd)
... Done [ OK ]
Waiting for web server at https://127.0.0.1:8443 to be available....
WARNING: web interface does not seem to be available!

I thought it was an issue with the web.conf, so I ripped that out and it still would not start. I then removed the SSL setting in the server.conf and the server started normally. I re-added the web.conf and it again restarted fine. It is a Comodo cert (with their weird chain). The certificate works find in the web interface. We use the same for both. I though maybe it wanted a password on the key file, so I added one, that did not help. Looking at splunkd.log I see:

05-17-2016 14:06:42.214 -0400 ERROR X509 - /opt/splunk/etc/auth/MYSERVER-01.key: unable to read X509 certificate file
05-17-2016 14:06:45.156 -0400 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/MYSERVER-01.key errno=185073780 error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch.

When the certificate is configured with the web interface, it passes all the verification checks.

[andrewcg · Accepted solution]

I finally figured it out. You have to put the certs together in this order (Comodo uses two intermediates):

1. Certificate
2. Password protected Key file
3. Intermediate #1
4. Intermediate #2
5. CA

into a file, for example: /opt/splunk/etc/auth/full.pem

Then assign that file to sslKeysfile in server.conf. No need to mess with the caCertFile setting.

[sslConfig]
sslKeysfile = full.pem
sslKeysfilePassword = XXXXXXXXXXX

This is totally different than the Splunk web settings where you leave the key in its own file and then combine the SSL certificate and the CA certs into one file.

[andrewcg]

An additional note, verify the full.pem before trying to restart. All the certificate BEGIN and END lines must be on their own line. For some reason the cert end and key begin line were always combining in my full.pem

-----END CERTIFICATE----------BEGIN RSA PRIVATE KEY-----

Splunk will not start with this in the full.pem. Just separate the lines:

-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----

and make sure you have 5 dashes at the beginning and end.