Deployment Architecture

Third Party server certificates not working- Is it not in the correct format?

parkertctr
Path Finder

My customers certificates expired and they followed the procedures for submitting and requesting a third party certificate.  The CA returned a CA certificate that was already combined. So the customer did not have to combine their certificates. When trying to start splunk, it will not start. When comparing all the certificates from previous ones, one thing we noticed was the private key had a heading "--BEGIN RSA PRIVATE KEY -- ", instead of "--BEGIN PRIVATE KEY--" and two new lines after, stating "Proc-Type" and "DEK-Info".  The customer is on Splunk v8.2.7, Windows 64bit.  The keys are DoD CA60

I am wondering if the private key is not in the correct format.  Should the customer re-submit a request to generate a new key from the CA?

Labels (2)
0 Karma
1 Solution

parkertctr
Path Finder

Thanks, we are good to go now. downloaded the new cert from the CA, received an error about encryption, removed the password and then fixed the absolute path in the web.conf.

View solution in original post

0 Karma

parkertctr
Path Finder

So far, we requested a new certificate from the CA. Tried again and still having issues.  Do we need to update the server.conf with the new path for the CA provided server certificates that are in the my certs directory? /etc/auth/mycerts/.... 

This is one of the error messages:

08-15-2022 14:12:19.559 -0500 WARN SSLCommon [19496 TelemetryMetricBuffer] - Received fatal SSL3 alert. ssl_state='error', alert_description='unknown CA'.

08-15-2022 14:12:19.838 -0500 ERROR X509Verify [19496 TelemetryMetricBuffer] - Server X509 certificate (CN=DoD WCF Intermediate CA 1,OU=WCF PKI,OU=DoD,O=U.S. Government,C=US) failed validation; error=20, reason="unable to get local issuer certificate"
 
0 Karma

PickleRick
SplunkTrust
SplunkTrust

As I wrote already - certificate handling (not just in Splunk but in general) can be a huge PITA. And Splunk seems a bit vague in docs about the cert files and forms. Anyway what worked for me was:

Get a .pem file with contents as follows:

<your server cert>
<your server's private key>
<intermediate CA cert>
[... <another intermediate CA in the path>...]
<root CA>

All of course in PEM format.

If your key or certificates are in different forms, convert them to PEM using appropriate tool (openssl?)

0 Karma

parkertctr
Path Finder

Thanks, we are good to go now. downloaded the new cert from the CA, received an error about encryption, removed the password and then fixed the absolute path in the web.conf.

0 Karma

parkertctr
Path Finder

We are receiving an error message "Received fatal SSL3 alert. ssl_state='error', alert_description='unknown CA'

 

0 Karma

parkertctr
Path Finder

Also, checked all the logs to see any errors related to ssl or certs. Nothing.. but splunk us starting

0 Karma

parkertctr
Path Finder

So it looks like splunk is starting but the web browser is not

0 Karma

parkertctr
Path Finder

Thanks. Yes I have been dealing with cert issues on another platform. The certs were in the proper formats. Let me double check if splunk start versus the web. We did check the logs and it did not have anything significantly sticking out regarding ssl. I will try again. more to follow. Hmm what do you recommend in checking if the certs are in PKCS#1 versus PKCS#8?

0 Karma

PickleRick
SplunkTrust
SplunkTrust

If I remember correctly, the "BEGIN PRIVATE KEY" format is PKCS#8 whereas "BEGIN RSA PRIVATE KEY" is PKCS#1.

Tags (1)
0 Karma

parkertctr
Path Finder

Tracking thanks

0 Karma

parkertctr
Path Finder

Working changing it. 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

BTW, I know it might be a silly question but just to be on the safe side - you did verify that your certificate matches the private key?

0 Karma

parkertctr
Path Finder

Yes everything matches. I think I am going to inform the customer to request a new cert from the CA and go from there. 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

As I said before - handling certificates often generates problems. The SSL error "unknown CA" suggests that something is not entirely right in the certificate chain.

The problem is we can't really give any specific advice without knowing the config and the certs. And you most probably can't show it to us and that's understandable.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

What do you mean by "will not start"? Typically (what is a bit annoying really) in case of certificate problems Splunk will happily start but ignore certs that are invalid from his point of view.

Anyway, firstly check the splunkd.log right after the start for anything cert-related.

Also remember that web.conf specifies cert and key files separately whereas in server.conf you're expected to provide combined key/cert file (specifying separate key file is supported but deprecated).

And indeed you might have to convert your key between PKCS#1 and PKCS#8 PEM encoded formats. (yes, handling certs and key files is a huge PITA)

0 Karma
Get Updates on the Splunk Community!

Earn a $35 Gift Card for Answering our Splunk Admins & App Developer Survey

Survey for Splunk Admins and App Developers is open now! | Earn a $35 gift card!      Hello there,  Splunk ...

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...