How to make my computer trust splunkd's own self-signed certificate?

[helderman]

I am developing a .NET application that uses Splunk.Client (https://github.com/splunk/splunk-sdk-csharp-pcl/tree/master/src/Splunk.Client) to query a Splunk instance.
Under the hood, Splunk.Client makes a REST API call to splunkd.
The HTTP request contains a Splunk query (e.g. index=main earliest=-1h), and splunkd responds with query results.
This works fine on our production environment.

For development and testing purposes, I installed a free edition of Splunk on my development PC (which, by the way, runs on Windows 10).
I configured the .NET application on my development PC to query my local Splunk instance.

Splunk comes out of the box with self-signed certificates.
Obviously these are not trusted by my computer, so Splunk.Client will throw an exception:

System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

I have little experience with certificates. I could just turn off certificate validation in the client (https://stackoverflow.com/questions/12506575/how-to-ignore-the-certificate-check-when-ssl), but I don't like the idea of a backdoor in production code.

I could go and buy a certificate, but that feels like overkill since it's just for development purposes.

I could make my own self-signed certificates, but that doesn't seem to bring any benefit over Splunk's self-signed certificates. There is a help document (https://docs.splunk.com/Documentation/Splunk/6.5.0/Security/Howtoself-signcertificates), but it's a dead end. The 'next steps' section has no reference to an article that explains how to make splunkd (port 8089) use the certificate. I edited Splunk's configuration files, but it seemed to have no effect. On this site I found questions concerning this issue, but hardly any useful feedback.

Splunk's self-signed certificate is good enough for me. What are the steps necessary to make my computer trust that certificate?

Disclaimer: I "don't have enough karma points to post links", hence the non-hyperlinks.

[pellegrini]

Only cacert.pem shall be placed in the certificate store. That file contains only the root certificate (public).

server.pem contains also the private server key and ca.pem contains also the private root key and that will compromise the security. Not that it really matters in this case since Splunk default root CA is used. In case of using a trusted signed cert, keep this in mind.

[helderman · Accepted solution]

I found an answer myself before posting the question. I posted it anyway; maybe somebody facing the same problem will find it useful.

The answer is based on Windows 10; much of this will be different in Linux, obviously.

Trouble is, the whole certificate validation is obscure by design, so you won't know what you've done wrong until you've fixed all issues and the validation succeeds. I hope the following solution covers it all, but I can't be sure. While struggling to get things done, I may have changed something not mentioned here, simply because it seemed irrelevant at the time.

1. Trust two certicates

In C:\Program Files\Splunk\etc\auth, there are two relevant certificates:

• server.pem: this appears to be the certificate used by splunkd
• ca.pem: this appears to be a root certificate that is necessary to get the certificate chain complete

Import both certificates into your computer's certificate store. Every certificate vendor will tell you how to. For Windows 10 instructions, just google 'mmc import certificate'.

There are a few pitfalls here:

• By default, the file browser of the Certificate Import Wizard filters on a number of known file extensions: CER, CRT, PFX... but not PEM. This seems to suggest PEM is not an accepted format. This is not true; just change the file type filter to 'All files (*.*)' and select the desired file.
• 'Automatically select the certificate store based on the type of certificate' seems like a convenient option in the Certificate Import Wizard, but it's not. I really had to specify the right certificate store myself to get things working. In the end, I had both certificates present in two different stores, which is probably more than strictly necessary, but it doesn't seem to hurt. Like I said earlier, I have little experience with certificates.
  • Trusted Root Certification Authorities
  • Intermediate Certification Authorities

2. Use the right host name

Look at the details of Splunk's self-signed certificate. You can either do this with a web browser (navigate to https://localhost:8089 and drill through the security warnings until you get to see the certificate) or from command line:

"C:\Program Files\Splunk\bin\splunk.exe" cmd openssl s_client -connect localhost:8089

Notice the certificate is issued to the following common name (CN): SplunkServerDefaultCert.

This name must match the host name in the URL you are using to access Splunk. https://localhost:8089 will not do; it must be https://SplunkServerDefaultCert:8089

It is possible to let your machine treat SplunkServerDefaultCert as an alias of localhost. The easiest way to accomplish this is to edit your hosts file. In Windows 10, this file is typically located in this folder: C:\Windows\System32\drivers\etc

Open the file in a text editor (e.g. Notepad++) and add the following line:

127.0.0.1       SplunkServerDefaultCert

Note: the editor must be running as administrator, otherwise you will not be able to save your changes.

In a web browser, verify the URL works: https://SplunkServerDefaultCert:8089

[kxnice]

@helderman

-- I was setting up HTTPs for Splunk and was struggling with understanding if i setup my CA's incorrectly, when it turned out the issue that was resulting in (https yellow lock), was the CN. You helped a lot and I thank you so much for your great knowledge and understanding in troubleshooting this issue!