Security

After upgrading to 6.4, why are our signed certs no longer accepted in server.conf?

andrewcg
Path Finder

After upgrading the to 6.4, Splunk web would no longer start:

Starting splunk server daemon (splunkd)
... Done [ OK ]
Waiting for web server at https://127.0.0.1:8443 to be available....
WARNING: web interface does not seem to be available!

I thought it was an issue with the web.conf, so I ripped that out and it still would not start. I then removed the SSL setting in the server.conf and the server started normally. I re-added the web.conf and it again restarted fine. It is a Comodo cert (with their weird chain). The certificate works find in the web interface. We use the same for both. I though maybe it wanted a password on the key file, so I added one, that did not help. Looking at splunkd.log I see:

05-17-2016 14:06:42.214 -0400 ERROR X509 - /opt/splunk/etc/auth/MYSERVER-01.key: unable to read X509 certificate file
05-17-2016 14:06:45.156 -0400 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/MYSERVER-01.key errno=185073780 error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch.

When the certificate is configured with the web interface, it passes all the verification checks.

1 Solution

andrewcg
Path Finder

I finally figured it out. You have to put the certs together in this order (Comodo uses two intermediates):

  1. Certificate
  2. Password protected Key file
  3. Intermediate #1
  4. Intermediate #2
  5. CA

into a file, for example: /opt/splunk/etc/auth/full.pem

Then assign that file to sslKeysfile in server.conf. No need to mess with the caCertFile setting.

[sslConfig]
sslKeysfile = full.pem
sslKeysfilePassword = XXXXXXXXXXX

This is totally different than the Splunk web settings where you leave the key in its own file and then combine the SSL certificate and the CA certs into one file.

View solution in original post

andrewcg
Path Finder

I finally figured it out. You have to put the certs together in this order (Comodo uses two intermediates):

  1. Certificate
  2. Password protected Key file
  3. Intermediate #1
  4. Intermediate #2
  5. CA

into a file, for example: /opt/splunk/etc/auth/full.pem

Then assign that file to sslKeysfile in server.conf. No need to mess with the caCertFile setting.

[sslConfig]
sslKeysfile = full.pem
sslKeysfilePassword = XXXXXXXXXXX

This is totally different than the Splunk web settings where you leave the key in its own file and then combine the SSL certificate and the CA certs into one file.

andrewcg
Path Finder

An additional note, verify the full.pem before trying to restart. All the certificate BEGIN and END lines must be on their own line. For some reason the cert end and key begin line were always combining in my full.pem

-----END CERTIFICATE----------BEGIN RSA PRIVATE KEY-----

Splunk will not start with this in the full.pem. Just separate the lines:

-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----

and make sure you have 5 dashes at the beginning and end.

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Announcing Modern Navigation: A New Era of Splunk User Experience

We are excited to introduce the Modern Navigation feature in the Splunk Platform, available to both cloud and ...

Kubernetes Alerts That Work Out of the Box with AutoDetect detectors in Splunk ...

A deployment rolls out slowly, one pod sits not ready for a few minutes, and whoever is on call gets an alert ...

Observability at Your Fingertips Directly From Splunk Platform—for Free

At .conf, we announced our roadmap to bring observability foundations to Splunk Platform. Bringing metrics and ...