When trying to apply my new SSL certificate to Splunk Web and restart, why does it just hang and the web server won't start?

[pdjhh]

Hi there,

When I try to apply my newly acquired SSL certificate to Splunk Web then restart, it just hangs and won't start the web server. Sequence of events was:

• I raised a csr and gave to the server guys to obtain a certificate.
• They provided a link from where I download the cert in x509 format as a .crt file with an intermediate certificate too.
• I combined these to into one pem file with the server cert at the top and intermediately underneath it as I read on Splunk website.
• I took the password off the original key and placed it and the new pem file into etc/auth/splunkweb
• I edited web.conf and pointed it at the key and cert files per docs
• Restarted Splunk Web, but couldn't connect, so I restarted Splunk and it runs through and hangs when trying to start the web server, indefinitely.
• Check the web service log and nothing.

Based on my steps have I done something wrong?

Thanks.

[nychawk]

Read here:
http://docs.splunk.com/Documentation/Splunk/6.2.5/Security/Getthird-partycertificatesforSplunkWeb

"Splunk Web does not support private key passwords"

You must to remove the passphrase altogether using an openssl command, then save the resulting certificate to a new one, pointing at that one.

Be sure to place your changes in "etc/system/local/web.conf"

Your new web.conf should look something like this:

[settings]
#
# SSL certificate files. Paths are specified relative to SPLUNK_HOME
#
privKeyPath = etc/auth/splunkweb/splunk-new-cert.key
caCertPath = etc/auth/splunkweb/splunk-new-cert.pem
#
enableSplunkWebSSL = true
startwebserver = 1
httpport = 8000
enableSplunkWebSSL = true

Be sure to follow best practices by disabling world read, write to these files, etc...

HTH,

-mi

[pdjhh]

Hi there. Thanks for posting that but that is actually what I've followed and I removed the passphrase from the key too. Are there any open ssl commands I can run to make sure the key and cert are all good and compatible with each other? The other thing I"m wondering about is the intermediate cert and if that's ok. I pasted it below the server cert as per instructions but not sure how to test it.

[grijhwani]

You did already mention removing the password in your original question.

You can validate the intermediate as you would any other certificate.

[grijhwani · Accepted solution]

You don't say what platform. If you're on Linux I would suggest using strace on a manually started Splunk instance to get straight into the problem.

Some thoughts though. Are you starting Splunk as a service (sudo /etc/init.d/splunk start or sudo service splunk start), or just invoking it manually as the splunk user. Have you checked permissions and ownerships. Remember that the key should only be root or splunk readable. If you're not running as root and it is root-owned that could cause problems. But it is odd that you're not seeing anything in the splunk log.

[pdjhh]

This behaviour, nothing happening when Splunk tries to reference the cert and key for Splunkweb and nothing in the logs, seems to be indicative of a problem with one or both of those from what I've read and people have advised. I went through the process from scratch; new key, new csr, new cert and loaded them up and all good.

Through this process I discovered some useful new tools, the csr checker to make sure all of your entries are correct and the matcher to check your new cert against the private key. This can be done in open ssl if you don't want to paste your private key onto the website, just search for 'verify cert and private key in openssl'.

https://cryptoreport.websecurity.symantec.com/checker/views/csrCheck.jsp

https://www.sslchecker.com/matcher

Thanks for guiding me in the right direction. Splunk didn't even answer the case I raised.

[grijhwani]

You're very welcome. Glad you got it sorted. That's what the community is for. I'm surprised the support team were non-responsive though.

[pdjhh]

I think I'm edging toward the problem. I put my certs and key into a checker and it came back and said the certificate and private key do not match. Having followed the steps on the Splunk website what are the circumstances under which this could happen? Anything to do with the organisation name and things like that I entered into the CSR when building that?

[grijhwani]

That was what I was getting at when I suggested you check the fingerprint of the key and the certificate. OpenSSL provides a commandline tool for generating the fingerprint of both the certificate and the key, and they should match. (This is what your checker will have done.)

As to how this comes about, quite simply the key and the certificate have been muddled. Either the Key you now have is not the key which against which the original CSR was created, or the certificate you have was not generated from the CSR you provided. My previous role required managing SSL certificates for multiple hosts and services, so there were standard processes for generating keys, CSRs and resultant certificates externally. When certificates were altered for Splunk I replaced them and altered the configuration to match manually. I never used Splunk to create the initial request, so I'm not sure where the procedure is that you followed, but an error like this essentially derives from human error.

Although it is configuration dependent (i.e. the local administrator can alter the location in web.conf), the default location for your web certificates is:
~splunk/etc/auth/splunkweb/privkey.pem
~splunk/etc/auth/splunkweb/cert.pem

I would search the whole of ~splunk/etc/ for .pem files, and check timestamps. You might be able to locate a match that way, if you can remember the date of generating the original request. Otherwise you will just have to generate a new key (or use an existing key), create a new CSR from it, and request a new certificate to match the new key. There is no feasible way to work backward if you cannot find the matching key for your certificate. (If there was SSL would not be secure...)

For a crash course in openssl certificate handling tools you could do worse than look at this summary

[pdjhh]

Hi there. Sorry yes it's Linux and in this case I'm just doing a splunk restart as the splunk user. I changed the ownership of the key and cert files to be the splunk user and both files currently have rw r r permissions.

[grijhwani]

What was the ownership and permission on the prior cert/key pair.

Did you confirm that the key/cert fingerprints actually matched, and that someone hasn't goofed in the cert generation?

And of course, there's always the possibility that if your Splunk server is normally started as a service, it will have been a root-owned process, and some of its prior working files will be root only. That can cause it major headaches if you then try to start it as the unprivileged splunk user.

[pdjhh]

I can see the fingerprint in the cert but not sure what to check that against? The rights and processes should be fine because always careful about that and the service has been running fine for months, it only broke now when I apply these. There must be something wrong with the certs and key combo or something.. MIght have to log a case and see if I can get some more help.

[grijhwani]

Fingerprint the key, too. That's what you compare with the certificate fingerprint.

You're absolutely certain that you have never invoked Splunk as root, and that there is nothing with the wrong file ownerships that might be stalling it?

Don't get caught up on the assumption that just because it was running when you stopped it that it would have restarted cleanly, unless having stopped it you then restarted it just to check. (This is, exactly what I do when I'm implementing a change, because experience has taught me it is easy to waste a lot of time getting funneled down a blind canyon by your assumptions.) It is quite possible that your certificate being the only thing you know to have changed is the problem, but don't forget that something else could have happened before that which would stop the process starting, even if it was fine while still running. If simple checks of your certificate setup do not solve the problem, then I would strongly suggest that you need to go to more direct methods to determine the definite fault rather than fixing what you think might be wrong.