Third Party server certificates not working- Is it not in the correct format?

[parkertctr]

My customers certificates expired and they followed the procedures for submitting and requesting a third party certificate.  The CA returned a CA certificate that was already combined. So the customer did not have to combine their certificates. When trying to start splunk, it will not start. When comparing all the certificates from previous ones, one thing we noticed was the private key had a heading "--BEGIN RSA PRIVATE KEY -- ", instead of "--BEGIN PRIVATE KEY--" and two new lines after, stating "Proc-Type" and "DEK-Info".  The customer is on Splunk v8.2.7, Windows 64bit.  The keys are DoD CA60

I am wondering if the private key is not in the correct format.  Should the customer re-submit a request to generate a new key from the CA?

[parkertctr]

So far, we requested a new certificate from the CA. Tried again and still having issues.  Do we need to update the server.conf with the new path for the CA provided server certificates that are in the my certs directory? /etc/auth/mycerts/.... 

This is one of the error messages:

08-15-2022 14:12:19.559 -0500 WARN SSLCommon [19496 TelemetryMetricBuffer] - Received fatal SSL3 alert. ssl_state='error', alert_description='unknown CA'.

08-15-2022 14:12:19.838 -0500 ERROR X509Verify [19496 TelemetryMetricBuffer] - Server X509 certificate (CN=DoD WCF Intermediate CA 1,OU=WCF PKI,OU=DoD,O=U.S. Government,C=US) failed validation; error=20, reason="unable to get local issuer certificate"

[PickleRick]

As I wrote already - certificate handling (not just in Splunk but in general) can be a huge PITA. And Splunk seems a bit vague in docs about the cert files and forms. Anyway what worked for me was:

Get a .pem file with contents as follows:

<your server cert>
<your server's private key>
<intermediate CA cert>
[... <another intermediate CA in the path>...]
<root CA>

All of course in PEM format.

If your key or certificates are in different forms, convert them to PEM using appropriate tool (openssl?)

[parkertctr · Accepted solution]

Thanks, we are good to go now. downloaded the new cert from the CA, received an error about encryption, removed the password and then fixed the absolute path in the web.conf.

[parkertctr]

We are receiving an error message "Received fatal SSL3 alert. ssl_state='error', alert_description='unknown CA'

[parkertctr]

Also, checked all the logs to see any errors related to ssl or certs. Nothing.. but splunk us starting

[parkertctr]

So it looks like splunk is starting but the web browser is not

[parkertctr]

Thanks. Yes I have been dealing with cert issues on another platform. The certs were in the proper formats. Let me double check if splunk start versus the web. We did check the logs and it did not have anything significantly sticking out regarding ssl. I will try again. more to follow. Hmm what do you recommend in checking if the certs are in PKCS#1 versus PKCS#8?

[PickleRick]

If I remember correctly, the "BEGIN PRIVATE KEY" format is PKCS#8 whereas "BEGIN RSA PRIVATE KEY" is PKCS#1.

[parkertctr]

Tracking thanks

[parkertctr]

Working changing it. 

[PickleRick]

BTW, I know it might be a silly question but just to be on the safe side - you did verify that your certificate matches the private key?

[parkertctr]

Yes everything matches. I think I am going to inform the customer to request a new cert from the CA and go from there. 

[PickleRick]

As I said before - handling certificates often generates problems. The SSL error "unknown CA" suggests that something is not entirely right in the certificate chain.

The problem is we can't really give any specific advice without knowing the config and the certs. And you most probably can't show it to us and that's understandable.

[PickleRick]

What do you mean by "will not start"? Typically (what is a bit annoying really) in case of certificate problems Splunk will happily start but ignore certs that are invalid from his point of view.

Anyway, firstly check the splunkd.log right after the start for anything cert-related.

Also remember that web.conf specifies cert and key files separately whereas in server.conf you're expected to provide combined key/cert file (specifying separate key file is supported but deprecated).

And indeed you might have to convert your key between PKCS#1 and PKCS#8 PEM encoded formats. (yes, handling certs and key files is a huge PITA)