Why am I having trouble with TLS?

[NJ]

Hi everyone.

I have followed the documentation for setting up TLS for inter-Splunk communication with self-signed certificates.

I have a small test environment that has an SH, an Indexer and an UF. 

However, I get the following error:

03-15-2023 01:23:39.475 +0000 ERROR TcpInputProc [2605538 FwdDataReceiverThread] - Error encountered for connection from src=10.0.0.4:45088. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

I have created the following certificates and keys based on the Splunk documentation: 

• myCertAuthCertificate.csr
• myCertAuthCertificate.pem
• myCertAuthCertificate.srl
• myCertAuthPrivateKey.key
• myServerCertificate.csr
• myServerCertificate.pem
• myServerPrivateKey.key
• mySplkCliCert.pem <- this is the concatenated file.

I copy the myCertAuthCertificate.pem and mySplkCliCert.pem files from the SH to the Indexer.

on the SH and Indexer, I edit the Server.conf to have the following:

[sslConfig]
sslRootCAPath = /opt/splunk/etc/auth/mycerts/myCertAuthCertificate.pem
serverCert = /opt/splunk/etc/auth/mycerts/mySplkCliCert.pem
sslPassword = *****

What am I doing wrong?

[michael_bates_1]

For the default configuration, ie no verification, then no certificate needs to be configured on the client end.

If you are looking to enable either client authentication (client must send a valid cert but its not checked beyond valid) or client verification (client provided certificate must match either CN or SAN) then you must define the client cert settings.

On the server side (in this case the indexer) the setttings in inputs.conf are

requireClientCert = <boolean>

 Client must provide a valid certificate, but its not checked

sslCommonNameToCheck = <comma-separated list>

sslAltNameToCheck = <comma-separated list>

Either of the above fill force the server end to actually fully validate the certificate against the identity of the host. 
e.g. if sslAltNameToCheck is set to splunkforwarder, then the certificate presented by the client must have the SAN field in the certificate set to include "splunkforwarder" otherwise the client is deemed to have failed authentication.

When getting to this level of verification, there are a few certificates that need to be distributed and configured to ensure that the Splunk instance is able to complete the verification all the way back to a trusted root CA.

[NJ]

Thank you so much for the details response @michael_bates_1 !!!

You have really helped my understanding!!!

[michael_bates_1]

The simplest analogy I can think of, is to simply consider this the same as the connection to a HTTPS web server.

Client - end initiating the connection
Server - Target of the connection.

For a basic SSL connection, only the server-side certificate is of any consequence.
The client certificate needs to be defined, but it is not used.

The client (sender) certificate only become relevant when you configure client verification on the server-side.

So, in this case you should be fine to use the same certificate on both ends of the SSL connection.

[PickleRick]

You're contradicting yourself. If client's certificate is not used, it doesn't have to be configured.

There are several different scenarios involving TLS but you'll probably want the client to authenticate itself - after all you don't want just anyone to send data to your splunk.

[NJ]

Thank you that makes a lot more sense for me now.

So to configure client verification I can create a new concatenated certificate that uses the same rootCAcertificate. Then tell the server to only accept connections that has a certificate that matches the new concatenated certificate. Is that correct?

[michael_bates_1]

No problems.

Essentially, until you are really comfortable with it, I always try and stick to the following for customers.
1. Unless you need to use organisational certificates, simply use the ones that Splunk generates. 
     You can easily regenerate these using a splunk command
      bin/splunk createssl
2. Start simple and work your way up - the further down this list, the more error prone and harder to troubleshoot
     a) default certs
     b) org certs
     c) server verification (client verifies the server cert)
     d) client verification  (server verifies the client cert)
3. Get familar with tools/commands like "openssl s_client" that can be used to verify the certificate exchange, etc.
    This can be really helpful to understand where the ssl protocol may be failing 
    e.g. openssl s_client -connect indexer:9997 -debug -showcerts 

[NJ]

Thanks, will keep that in mind!

Can you please just confirm for me that the following is correct:

On both the client and the server I use the same concatenated certificate or do i need to create one concatenated certificate for the server and one for each client?

[michael_bates_1]

If it is being done to meet an org requirement, then it looks like the whole certificate process may need to be re-visited

[michael_bates_1]

Hmm. Looks like there is something wrong with the cert.
Are you creating your own cert due to organisation requirements or for some other reason.
If you are crating "just because" then I might suggest just using the certs and config from earlier that worked.

If it is being done to meet an org requirement, then it looks like the whol

[NJ]

Hi @michael_bates_1 

Really appreciate your help so far!!!

I need to understand the process of setting up TLS so I can do it for future customers. 

[michael_bates_1]

If you search index=_internal x509 since the last restart of the indexer, does anything show up

[NJ]

03-16-2023 02:33:18.457 +0000 ERROR X509 [1969765 TcpChannelThread] - X509 certificate (CN=splunkbase.splunk.com,O=Splunk Inc.,L=San Francisco,ST=California,C=US) common name (splunkbase.splunk.com) did not match any allowed names (apps.splunk.com,cdn.apps.splunk.com)
host = Splunk-SearchHead-RedHat1source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd

03-16-2023 03:01:02.758 +0000 ERROR X509 [2841646 ApplicationUpdateThread] - X509 certificate (CN=splunkbase.splunk.com,O=Splunk Inc.,L=San Francisco,ST=California,C=US) common name (splunkbase.splunk.com) did not match any allowed names (apps.splunk.com,cdn.apps.splunk.com)
host = Splunk-Indexer-Redhat1source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd

[michael_bates_1]

This indicates that the issue is likely to be certificate or file permissions related.
As a side note, not sure why you have the server cert on the client end.

Can you please confirm the certificate you are trying to configure on the server side (inputs.conf) includes all the certificates in the chain that would be required to fully verify (host all the way back to root ca).

[NJ]

I have found the Splunk documentation quite confusing regarding TLS.

When creating the keys, I followed the same naming conventions used in the Splunk documentation: How to create and sign your own TLS certificates - Splunk Documentation

Which gave me the following files:

• myCertAuthCertificate.csr
• myCertAuthCertificate.pem
• myCertAuthCertificate.srl
• myCertAuthPrivateKey.key
• myServerCertificate.csr
• myServerCertificate.pem
• myServerPrivateKey.key

I then did the following command

cat myServerCertificate.pem myServerPrivateKey.key myCertAuthCertificate.pem > mySplkCliCert.pem

In inputs.conf I have the concatenated cert (mySplkCliCert.pem)

[michael_bates_1]

This should work, as a test, place each part on the required element

outputs.conf
---------
[tcpout]
defaultGroup = indexers
indexAndForward = false

[tcpout:indexers]
server = x:9997
useSSL = true
sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
sslPassword = password

inputs.conf
-----
[splunktcp-ssl:9997]
disabled = 0

[SSL]
serverCert = $SPLUNK_HOME/etc/auth/server.pem
sslPassword = password
requireClientCert = false

[NJ]

Thats works! 

[michael_bates_1]

Change both ends just to be sure.

[NJ]

Should inputs.conf have serverCert = /opt/splunk/etc/auth/server.pem

if yes then what should outputs.conf have in sslCertPath = /opt/splunk/etc/auth/?

[michael_bates_1]

Are you able to revert all elements to use the default certificate
/etc/auth/server.pem

You can also check the internal logs (index=_internal) and look for x509
This will normally on appear after a reboot.

[NJ]

So i should change the output.conf to use the /opt/splunk/etc/auth/server.pem?