What are digital certificates and their trust chains?

[dtsariapkin]

Greetings Splunk Community, 

This will be one of the several community/answers posts in regards to the SSL overall and how we do it in Splunk. It will be an authors explanation of digital certificates and their trust relationships. 

Disclaimer

I do not claim to be the best expert on digital certificates. I hope the explanation that way will make it easier for everyone to grasp the concept and help with any level of troubleshooting of digital certificates involved.

1. It all begins with Asymmetric encryption.

To start the conversation on digital certificate one must be comfortable with basics of encryption. For a quick recap encryption mainly utilizes a key.

Encryption Key - An encryption key is a string of bits (data) generated specifically to scramble and unscramble data. So you use a key to produce Encrypted text (cipher text). Or to Decrypt what you have encrypted before.

There are primarily 2 types of encryption used for data transfer:
- Symmetric - that is when your key is the same on both ends and you encrypt and decrypt data the same way. Most easy to use and provides the best performance (throughput and resources used to do).

- Asymmetric - that is when you have a pair of 2 keys. Key 1 used for Encrypting data and Key 2 used for Decrypting.

Hope I did not lose you so far?

2. Asymmetric is AWESOME! But why not use it everywhere?

So Asymmetric encryption is awesome. But it’s terribly resource greedy and provides a huge overhead. Why? well when you encrypt using it you provide a much bigger output then what you put into it. So it makes no sense at current stage of development of technology to use it in your large bulky data transfer.

So we use asymmetric encryption a lot but for two absolutely different use cases:

1. When you need to provide data that anyone can read. But none can really change it. How? Easy. Give everyone Part of your Asymmetric key which they can use for decryption. But do not give the encryption key. That way anyone can decrypt what you show them. But none can re-encrypt the data to pretend to be You. In short that’s what you can hear or refer to be “Digital Signature”
2. The other way round. You give everyone your key for Encrypting the data, and then keep your key for Decrypting it to be very private. What is that used for? Easy! For establishing the secure connection using the symmetric key. TL:DR: send everyone your public key. Now they can generate their own Symmetric key AND then send it to you Encrypted with your key. And none can De-crypt it, because only you have that key.

3. And that brings us to digital certificates and Trust

So what we have learned so far brought us right to certificates.

Digital Certificates - are a combination of both things discussed above. They contain the Public → encryption key of the server you are connecting too. AND! Ideally they are “Digitally Signed” by someone you Trusted beforehand (or not)

Important note! (see Spoiler below):

Spoiler

Digital certificates not necessary have to be trusted. In browser you would then have a red warning and can just say “proceed anyway” or in Splunk we configure:

• sslVerifyServerCert = <boolean>

Which is false by default. This can be seen in several configurations files depending on the connection usage. Examples are:

• Global setting in server.conf [sslConfig] stanza.

• Forwarder to Deployment server specific connection deploymentclient.conf

• Forwarder to Indexer connection outputs.conf

In a browser You see this for not trusted:

But what is Trust then?

And that’s a very good question. So in every software in the world which uses digital certificates there’s a certain collection of certificates that is either built-in or was configured by admins.

Allow me to introduce some more concepts of Trust Store and Self-Signed.

Trust Store - is a collection of Certificates which have the asymmetric keys in them. Which you or administrators have configured as coming from trusted sources. Most of the time issued by Servers to themselves so they are signed by themselves → hence the term Self-Signed

The moment you add a certificate to a trusted store. It means → whatever the holder of this Asymmetric key has Digitally Signed (still remember that part?) now you trust that to be true and good for you. That they not gonna harm you.

You get this kind of a green (or lock) for Web Sites that are good/trusted:

To have that that configured in your conventional operating systems for example:

• your Windows would have a Trust store (Certificate Manager) → certmgr.msc

• And your Apple Mac OS would have trust store:

AND? You guessed it. In Splunk we also configure somewhat of a trust store. BUT! For each component separately. Most common use in server.conf :

[sslConfig]
sslRootCAPath = <path>
* Full path to the root CA (Certificate Authority) certificate store
  on the operating system.
* The <path> must refer to a PEM (Privacy-Enhanced Mail) format
  file containing one or more root CA certificates concatenated
  together.
* Required for Common Criteria.
* This setting is valid on Windows machines only if you have not set
  'sslRootCAPathHonoredOnWindows' to "false".
* No default.

 I will cover Splunky way in a bit more detail of chapter 5 of this article. 

4. Tying it all up in Trust chains

Now that you are familiar with concept of certs and where we store so called “trusted” certs. Let’s explore further.

Let’s see what are trust relationships so to say. So because each trusted store is stored individually each machine decided who it trusts absolutely independently. And that’s the thing you must always remember. Each machine builds it’s own trust chain and trusts independently and checks certificates only when it receives them from someone as to whether they are trusted or not.

Now we go further. Trust chains and how do we check if something is trusted. Each machine running some kind of SSL/use of certificate connection will have it’s own certificate → Identity Certificate

In Splunk Identity Certificate is the one you configure as serverCert in server.conf:

serverCert = <path>
* The full path to the PEM (Privacy-Enhanced Mail) format server
  certificate file.

So when we connect to any server using say HTTPS we will see this machine's identity certificate AND!! it will be digitally signed by someone we trust. So it will be looking like this:

• Subject → who is the machine who had the cert (server you are connecting to)

• Certificate Authority → who signed the certificate for you

Now the most interesting part to tie it all up:

The computer (for example machine running Splunk Forwarder) tries to connect to a server (Deployment server for example or Indexer)

• Server to which you are connecting to will introduce itself with certificate. IMPORTANT! and now originating machine (Forwarder in this example) will be building it’s trust chain.

Yes it is like that. And no matter what. To trust or not to trust will be decided on what we call client side of the connection → that is machine which originates the connection.

Since we now got server’s Identity Certificate - that’s what it’s usually called as that’s how server will identify itself. There are 2 ways to check it’s trust depending on the software itself.

1. The difficult way of matching the certificates or AKI to SKI → Authority Key Identifier to Subject Key Identifier I will cover in separate article.

2. The easy (lazy) way → Check who issued the certificate by name. Look in this machine's trusted store for the issuer field of the certificate. And something just that simple you just check by the Name of the issuer and that’s it like so:

5. Splunk way of trust in Trust chains

Now that you know all of the above let's get into Splunk. Splunk takes it's certs in PEM format:

PEM (originally “Privacy Enhanced Mail”) is the most common format for X. 509 certificates, CSRs, and cryptographic keys. 

If you open your PEM certificate in any text editor you going to see something like that:

-----BEGIN CERTIFICATE-----
MIIDejCCAmICCQCNHBN8tj/FwzANBgkqhkiG9w0BAQsFADB/MQswCQYDVQQGEwJV
UzELMAkGA1UECAwCQ0ExFjAUBgNVBAcMDVNhbiBGcmFuY2lzY28xDzANBgNVBAoM
BlNwbHVuazEXMBUGA1UEAwwOU3BsdW5rQ29tbW9uQ0ExITAfBgkqhkiG9w0BCQEW
EnN1cHBvcnRAc3BsdW5rLmNvbTAeFw0xNzAxMzAyMDI2NTRaFw0yNzAxMjgyMDI2
NTRaMH8xCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTEWMBQGA1UEBwwNU2FuIEZy
YW5jaXNjbzEPMA0GA1UECgwGU3BsdW5rMRcwFQYDVQQDDA5TcGx1bmtDb21tb25D
QTEhMB8GCSqGSIb3DQEJARYSc3VwcG9ydEBzcGx1bmsuY29tMIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzB9ltVEGk73QvPlxXtA0qMW/SLDQlQMFJ/C/
tXRVJdQsmcW4WsaETteeWZh8AgozO1LqOa3I6UmrWLcv4LmUAh/T3iZWXzHLIqFN
WLSVU+2g0Xkn43xSgQEPSvEK1NqZRZv1SWvx3+oGHgu03AZrqTj0HyLujqUDARFX
sRvBPW/VfDkomHj9b8IuK3qOUwQtIOUr+oKx1tM1J7VNN5NflLw9NdHtlfblw0Ys
5xI5Qxu3rcCxkKQuwz9KRe4iijOIRMAKX28pbakxU9Nk38Ac3PNadgIk0s7R829k
980sqGWkd06+C17OxgjpQbvLOR20FtmQybttUsXGR7Bp07YStwIDAQABMA0GCSqG
SIb3DQEBCwUAA4IBAQCxhQd6KXP2VzK2cwAqdK74bGwl5WnvsyqdPWkdANiKksr4
ZybJZNfdfRso3fA2oK1R8i5Ca8LK3V/UuAsXvG6/ikJtWsJ9jf+eYLou8lS6NVJO
xDN/gxPcHrhToGqi1wfPwDQrNVofZcuQNklcdgZ1+XVuotfTCOXHrRoNmZX+HgkY
gEtPG+r1VwSFowfYqyFXQ5CUeRa3JB7/ObF15WfGUYplbd3wQz/M3PLNKLvz5a1z
LMNXDwN5Pvyb2epyO8LPJu4dGTB4jOGpYLUjG1UUqJo9Oa6D99rv6sId+8qjERtl
ZZc1oaC0PKSzBmq+TpbR27B8Zra3gpoA+gavdRZj
-----END CERTIFICATE-----

This is a text representation of a cert. But what interests you is that it has beginning and ending of the cert. 

But that's just one! For a trust chain you need more then one. More so when you have several different certificates that all need to be trusted. 

In Splunk we rely on the certificates being present in single pem file mentioned in configurations before server.conf . As of time of this article there are several other configurations available depending on the purpose for inputs and for deployment client but the fallback is always to sslRootCAPath. 

How do we do it? It's that simple! just take your certs in text format and copy paste into a single file. Important note include beginning and end. Like so:

-----BEGIN CERTIFICATE-----
... (certificate for your server)...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
... (the intermediate certificate)...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
... (the certificate authority certificate)...
-----END CERTIFICATE-----

Splunk’s official document link here:

How to prepare TLS certificates for use with the Splunk platform - Splunk Documentation

This you can do in any text editor (e.g. notepad, notepad++, nano, vim, etc.). Just keep adding certificates as needed to the same file. That will create your trusted store that we talked about before. A single file where you store chains to trust. 

You can also use an easy *nix way of doing that:

cat myServerCertificate.pem myIntermidiateServer.pem myCACertificate.pem > Combined.pem

In this example it just reads all 3 files and then you output that to another file ( > means output instead of screen somewhere else)

Spoiler

Note!! As of the time of this article. The certs can be pasted in any order and still work.

If you read it in just normal openssl x509 command it will show you just one cert.  To view all certificates in one PEM file use openssl command like example below. You can add -text parameter if you want to see more of the expiration date on certs and so on.

openssl crl2pkcs7 -nocrl -certfile Combined.pem | openssl pkcs7 -print_certs -noout

subject=/C=NL/postalCode=5705 CN/L=City/street=Example 20/O=Foobar B.V./OU=ICT/OU=Wildcard SSL/CN=*.example.com
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Organization Validation Secure Server CA

subject=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Organization Validation Secure Server CA
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority

[lanzykevin]

I am lost on one part, what would the cert needed for the "sslRootCAPath"? Would this be the Root cert from the CA? I am so confused on this part.

[dtsariapkin]

hi @lanzykevin Great question!! 

So this part covers it a bit. I will paste it below. In more explained way though. What you would need is depending on your Issuing chain. If you got only Single CA then add it's root certificate. BUT! You most probably in most cases would have Root CA -> Intermidiate CA -> Possibly one more CA -> 

Then you would need to take each of those certificates and concatenate them all to this chain that I describe. Does that make sense? 

From above text see this:

This you can do in any text editor (e.g. notepad, notepad++, nano, vim, etc.). Just keep adding certificates as needed to the same file. That will create your trusted store that we talked about before. A single file where you store chains to trust. 

You can also use an easy *nix way of doing that:

cat myServerCertificate.pem myIntermidiateServer.pem myCACertificate.pem > Combined.pem

[lanzykevin]

@dtsariapkin

   My understanding is then that the "server.pem" cert I have I would cat that with my "CAcert.pem" cert? I think I may be skipping back and forth on steps in the process. I already have everything for the "serverCert" stanza. Used an internal CA to get that done.

What I am gathering is that I would need to do, almost, the same process but with the rootCA part? I have the root CA so I would cat that with the "server.pem" cert and that would be it?

I'm sure this seems trivial but I am wracking my brain here trying to understand this.

[dtsariapkin]

This article is meant for many uses cases. Not just singular. It should give the good understanding about how certificates work overall. 

Just like sslRootCAPath is meant to TRUST the certificates coming from other instances. I will give you some examples:

- License Manager connection

- Heavy forwarder to Deployment Server connection

- Cluster Manager connection.

etc. Please give it a whole thorough read once again top to bottom to understand the inner workings of Certificates. And then apply it accordingly to your specific need, architecture and deployment. 

P.s. server.pem is something I discuss in this article too. As the Identity Certificate. What you say is true "I have the root CA so I would cat that with the "server.pem" cert and that would be it?" but that would be just for the indeitity cert itself. 

Once you get into the weeds of who sends certificate to who. As in e.g. -> 
- When indexers connect to License manager they would get certificate from LM being server certificate. 

- But then You can also expand that knowledge now. And apply that to the Universal Forwarder. There you have something you can Client Cert. Which also would need to be trusted on the other side. 
- OR forwarder connection to Deployment Server. Deployment Server needs to have server cert right? But then Universal Forwarders need to trust it too IF you do verify server cert there. 

And the list goes on. 

TL:DR 
Server.pem -> in most cases it's just the:
- Public key inside certificate of the server

- Private key corresponding to the public key above

- Certificate of whoever issued it. 

sslRootCAPath -> Can be any number of certificates depending on the type of connections you use it for. And the number of different CAs you might have. You might be a middle man with several customers connecting to you each using their own certs and you need them trusted. 

Hope that makes more sense now. 

[dtsariapkin · Accepted solution]

And that is really it! You can have chain of more then two certificates. Or add more certificates to trust into the same file.

Spoiler

Example!! on how the connection will work with Splunk Universal forwarder connecting to Indexer:
UF sends connection request -> IDX responds with Cert in server.pem configured as serverCert -> UF Checks the certificate by using it's Store - sslRootCAPath

Though it all relies on your understanding of certificates and their trust relationships. And some administration of Splunk to configure it.

Important reminder! Certificate chains are checked and built locally by each machine so it can look differently depending on the trusted store on the machine as certificate expire and change over time. Especially during troubleshooting check in case of differences between machines if all else fails.

Useful commands

• /opt/splunk/bin/splunk cmd openssl s_client -showcerts -connect 10.20.30.40:8089
• openssl x509 -text -noout -in cert1.pem
• openssl crl2pkcs7 -nocrl -certfile combined.pem | openssl pkcs7 -print_certs -text -noout
• /opt/splunk/bin/splunk cmd openssl s_client -showcerts -connect 10.20.30.40:8089 -CAfile /opt/splunk/etc/auth/MyCombined.pem