Proper way of using a certificate chain

[echalex]

Hi,
I'm trying to set authenticate the forwarders using SSL certificates. If using certificates signed by a single root CA, this causes no problems. However, when using an intermediate CA, I can't seem to get it working no matter what. I'm using Splunk 4.2.3.

I'm basing my efforts on this:
http://www.splunk.com/wiki/Community:Splunk2Splunk_SSL_3rdPartyCA

Q1) Which certificates need to be stored on the forwarder and indexer? I'm guessing both.

Q2) Where should these certificates be located? Does order matter? Currently, I'm putting them both in cacert.pem. I've tried both putting the root CA first and the intermediate CA first. If I put the root CA first, the indexer will complain that the CA is untrusted. If I put the intermediate CA first, the indexer will report "routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned"

Q3) On both the forwarder and the indexer, I've done this:
"cat cert.pem key.pem cacert.pem >server.pem". (Based on the document linked to above.) Is this correct?

forwarder's system/local/outputs.conf

[tcpout]

defaultGroup = default-autolb-group

disabled = false

[tcpout-server://indexer1.company.com:9996]
sslCertPath = /opt/splunkforwarder/etc/auth/server.pem
sslPassword = wouldntyouliketoknow
sslRootCAPath = /opt/splunkforwarder/etc/auth/cacert.pem

[tcpout:default-autolb-group]
autoLB = true
disabled = false
server = indexer1.company.com:9996,indexer2.company.com:9996

[tcpout-server://indexer2.company.com:9996]
sslCertPath = /opt/splunkforwarder/etc/auth/server.pem
sslPassword = wouldntyouliketoknow
sslRootCAPath = /opt/splunkforwarder/etc/auth/cacert.pem

indexer's system/local/inputs.conf:

[default]

host = indexer1.company.com

[splunktcp-ssl:9996]

[SSL]
password = betyoudliketoknow
requireClientCert = true
rootCA = /opt/splunk/etc/auth/cacert.pem
serverCert = /opt/splunk/etc/auth/server.pem

Any tips would be greatly appreciated.

[echalex · Accepted solution]

This issue was solved after installing Splunk 4.3. For one thing, one has to make sure that the certificates are chained correctly, as tross33 pointed out.

However, the real problem was that the certificate used in the forwarders had the purpose-field set to server. The SSL library doesn't want to accept that purpose for client certificates. The solution is to make sure the certificate is either a client certificate or multi-purpose. This must be done by the CA when signing.

Installing Splunk 4.3 helped, since it improved the error reporting on SSL problems:

For info on a certificate's supported purposes, you can use the openssl utility:

foo@bar:~/certs $ openssl x509 -purpose  -noout -in certificate.pem
Certificate purposes:
SSL client : Yes
SSL client CA : No
SSL server : Yes
SSL server CA : No
Netscape SSL server : Yes
Netscape SSL server CA : No
S/MIME signing : Yes (WARNING code=2)
S/MIME signing CA : No
S/MIME encryption : Yes (WARNING code=2)
S/MIME encryption CA : No
CRL signing : Yes
CRL signing CA : No
Any Purpose : Yes
Any Purpose CA : Yes
OCSP helper : Yes
OCSP helper CA : No
Time Stamp signing : No
Time Stamp signing CA : No

[trross33]

One thing I noticed. You mentioned:

Q3) On both the forwarder and the indexer, I've done this:
"cat cert.pem key.pem cacert.pem >server.pem". (Based on the document linked to above.) Is this correct?

You might want to try:
"cat cert.pem key.pem intermediatecacert.pem rootcacert.pem > server.pem"

[echalex]

I'm commenting a bit late, I know, but...
In effect, that's what I've done, tross33. You see the cacert.pem has been created by doing

cat intermediatecacert.pem rootcacert.pem >cacert.pem